How to auto scale Apache Kafka with Tiered Storage in Production

In the distributed systems world, Apache Kafka has earned its reputation as the gold standard for scalability. Companies routinely run Kafka clusters with tens or even hundreds of brokers, handling massive throughput with remarkable stability. Yet for all that scalability, Kafka has always struggled with one critical part of modern infrastructure: elasticity.

The OSO engineers have been working on this problem, and tiered storage is finally making Kafka meaningfully elastic. It’s not just a theoretical improvement. It changes the economics and feasibility of reactive scaling in production.

The distinction matters more than you might think. Scalability lets Kafka grow gracefully as your business expands over months and years. Elasticity lets it respond to traffic spikes within minutes, then scale back down when demand subsides. For an infrastructure component that is often one of the largest compute costs in a data platform, the difference has real operational and financial impact.

All code for this post is in the kafka-auto-scaling-example repository on GitHub.

Updated October 2026: tiered storage status, configuration names and limitations have been checked against the Apache Kafka 3.9 release notes and 4.3 tiered storage documentation, and the Kubernetes scaling steps against the current Strimzi documentation.

Why does Kafka scale out but struggle to scale down?

Kafka’s scaling success story is well documented. A startup can begin with three brokers and grow to enterprise scale without architectural changes. The core design, topics split into partitions and partitions replicated across brokers, handles that growth well. When you need more capacity, you add brokers, create new topics with more partitions and spread the load.

That organic growth works because it’s planned and gradual. Teams add capacity before they need it, new topics naturally spread across the bigger cluster, and the system grows with the business.

Elasticity is a different challenge. Imagine a hype-driven traffic spike: your product hits the top of Reddit, or a marketing campaign beats expectations by 10x. You need capacity now, not next week. Traditional scaling fails here because of a basic architectural constraint: partition replicas have fixed assignments to brokers.

When you scale a three-broker cluster to four, the new broker starts empty. Existing partition replicas stay on the original three brokers, so there is no performance benefit until data moves. The new broker sits idle, using resources while contributing nothing to throughput.

Scale-down is even harder. If you remove a broker that still hosts partition replicas, you either lose redundancy and availability or postpone scaling until you can safely move the data elsewhere.

This is what separates Kafka from stateless applications, where scaling means adding or removing identical instances. Each broker holds unique state: its partition replicas, including replicas of the internal topics that store committed consumer offsets and transaction state. You can’t terminate a broker the way you terminate a web server.

The answer is data movement, and that’s where the complexity starts. To make a new broker useful, you rebalance partition replicas across the cluster. To remove a broker safely, you first move its replicas to the remaining brokers. Both operations are resource-intensive and slow.

What does traditional Kafka scaling really cost?

The OSO engineers have seen the resource impact of traditional Kafka scaling across many enterprise deployments, and the numbers are sobering. Take a modest example: scaling from three brokers to four, where each original broker holds 4 TB. Reaching balance means moving about 3 TB from the existing brokers to the new one.

That movement eats network bandwidth. In the cloud, transfer between instances in the same zone may be free, but the bandwidth itself becomes the bottleneck. Source brokers read from disk while the destination writes, and CPU climbs with compression, decompression and TLS overhead.

The result is a performance paradox: scaling up first makes the cluster slower, because resources are diverted to moving data. Instead of gaining capacity immediately, you temporarily lose it until the rebalance completes.

The rebalance can’t take over the cluster either, because production traffic has to keep flowing. Reassignments are throttled to protect live workloads, which stretches them out further. Moving several terabytes while keeping production performance can take hours or days, far too slow for spikes that last minutes or hours.

That forces an uncomfortable trade-off: run with significant over-capacity to absorb spikes, or accept that scaling can’t respond to sudden demand.

How does tiered storage make Kafka elastic?

Kafka tiered storage, introduced as early access in Kafka 3.6 and declared production-ready in Kafka 3.9, changes the equation. Brokers keep recent “hot” segments on local disk and move completed older segments to a remote tier such as Amazon S3, which dramatically reduces how much data each broker stores locally.

Take the same scaling scenario with tiered storage. Instead of 4 TB per broker on local disk, you might have 400 GB of hot data per broker and 3.6 TB in object storage. When scaling up, a new replica only copies the local portion of each partition; older segments are already in the remote tier. That’s roughly a 10x reduction in data movement.

The improvement cascades through the whole operation. Network transfer and disk I/O drop by around 90%, and rebalancing time falls from hours to minutes. Suddenly, reactive scaling is practical.

Tiered storage is off by default. You enable it on the brokers with a remote storage plugin, then per topic. Apache Kafka doesn’t ship a RemoteStorageManager implementation, so you supply one (for example an open-source S3 plugin):

remote.log.storage.system.enable=true
remote.log.storage.manager.class.name=<RemoteStorageManager class from your plugin>
remote.log.storage.manager.class.path=/opt/kafka/plugins/tiered-storage/*

kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders \
  --partitions 12 --replication-factor 3 \
  --config remote.storage.enable=true \
  --config local.retention.ms=21600000 \
  --config retention.ms=2592000000

local.retention.ms and local.retention.bytes control how much stays on the broker; retention.ms and retention.bytes still control total retention across both tiers.

Object storage economics

Object storage brings benefits beyond scaling speed. Services like Amazon S3 offer practically unlimited capacity at a lower cost per GB than block storage, which removes the capacity planning that often forces you to over-provision broker disks.

The pricing model also fits real usage better: you pay for what you store, plus request charges for access. For rarely read data, that usually costs much less than the same capacity on local block storage.

The economics get more complicated when cold data is read often. Applications that repeatedly read old data can run up request charges, so tiered storage works best when cold data access really is archival.

When tiered storage doesn’t fit

Tiered storage isn’t for every workload. Some use cases need all data locally with minimal latency, and real-time analytics, fraud detection or some stream processing may not tolerate the higher latency of reading old data from object storage. There are also hard limits in the Kafka documentation: compacted topics aren’t supported, and tiered storage has to be disabled on every topic before it can be disabled on the brokers.

The OSO engineers recommend analysing your access patterns before enabling it. The best candidates have clear hot and cold boundaries: recent data read often, older data read rarely or never.

How do you auto scale Kafka on Kubernetes?

Modern Kafka increasingly runs on Kubernetes, and the operator pattern gives you orchestration that understands Kafka rather than just scaling pods. Strimzi is the most widely used open-source example.

In Strimzi, brokers belong to a KafkaNodePool, and you scale by changing its spec.replicas, for example with kubectl scale kafkanodepool pool-a --replicas=4. Because node pools support the Kubernetes scale operation, an autoscaler or your own automation can drive the replica count, while the operator handles the Kafka-specific work. It doesn’t just create pods: it coordinates broker startup, cluster membership and data rebalancing in the right order.

Order matters. New brokers must fully join the cluster before rebalancing starts, and rebalancing must finish before the cluster counts as scaled. Scale-down needs even more care. Strimzi blocks removing a broker that still hosts partition replicas, so data has to move off first. You can bypass that check with an annotation, but doing so risks failed reconciliation and lost replicas:

kubectl annotate kafka my-cluster strimzi.io/skip-broker-scaledown-check="true"

Cruise Control auto-rebalancing

Cruise Control, originally built at LinkedIn and now maintained as an open-source project, provides the rebalancing algorithms production auto-scaling needs. Rather than naive round-robin placement, it weighs CPU, disk, network I/O, rack awareness and leader distribution.

Strimzi can trigger Cruise Control automatically on scaling events. With autoRebalance configured, partitions are spread onto new brokers after a scale-up (add-brokers mode) and moved off brokers before a scale-down (remove-brokers mode):

apiVersion: kafka.strimzi.io/v1
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    # ...
  cruiseControl:
    autoRebalance:
      - mode: add-brokers
      - mode: remove-brokers

Only changes to spec.replicas on an existing node pool count as scaling events; creating or deleting a node pool doesn’t trigger auto-rebalancing. You can point each mode at a KafkaRebalance template to tune goals for your workload rather than just equalising data.

Which metric should trigger scaling?

CPU is the simplest scaling metric, but Kafka-specific metrics often make better decisions. Request handler and network thread idle ratios, request queue sizes and consumer lag can show capacity limits before CPU does.

The OSO engineers find that the best metric depends on the workload. Producer-heavy workloads may scale best on network thread utilisation, consumer-heavy ones on fetch request latency, and storage-heavy ones on disk usage. A Prometheus-based stack, for example through a metrics adapter or KEDA, can feed these Kafka metrics to the Kubernetes autoscaler.

When should you use Kafka auto-scaling in production?

Even with these improvements, Kafka auto-scaling works best for predictable patterns rather than emergency capacity. The OSO engineers find it most effective for:

• Daily patterns, scaling up for business hours and down overnight
• Weekly patterns, scaling down at weekends
• Seasonal adjustments, scaling gradually ahead of expected growth

These give scaling operations time to finish and settle before peak demand. Automation removes the toil of manual scaling while keeping the planned approach that suits Kafka.

For unpredictable spikes, such as viral content, a surprise marketing hit or breaking news, scaling manually ahead of expected demand is still the most reliable approach. Auto-scaling can’t add capacity fast enough for sudden spikes.

Kubernetes capacity planning

Kubernetes adds another layer. When the autoscaler adds broker pods, they need compute. If the cluster has no spare capacity, pods stay pending while the cluster autoscaler provisions nodes.

Cloud node provisioning typically takes 2 to 5 minutes, plus time for the node to join, the pod to schedule and the broker to start. That delay adds to the Kafka scaling time and makes reactive scaling even less practical for sudden spikes. The OSO engineers recommend keeping buffer capacity in the Kubernetes cluster so broker pods schedule immediately, trading some efficiency for responsiveness.

Stabilisation windows

Auto-scaling must avoid oscillation, scaling up and down repeatedly on short metric swings. This matters more for Kafka because scaling itself changes the metrics: during a scale-up rebalance, CPU and network spike as data moves. Without stabilisation windows, that spike could trigger another scale-up and destabilise the cluster.

The OSO engineers typically use stabilisation windows of 10 to 15 minutes for Kafka, longer than the Kubernetes default of 5 minutes for scale-down and none for scale-up, so metrics settle after each operation. That prevents loops while staying responsive to sustained load changes.

How should you implement Kafka auto-scaling?

Based on production experience, the OSO engineers recommend this approach:

1. Start with a tiered storage assessment. Analyse access patterns to understand your hot and cold data ratio, and check you have no compacted topics you expected to tier. Measure how much local storage you’d save to estimate the scaling improvement.
2. Configure conservative autoscaler settings. Begin with long stabilisation windows (15 minutes or more) and shorten them based on what you observe. Scale-up policies can be more aggressive than scale-down, because adding capacity is safer than removing it.
3. Use auto-scaling for predictable patterns. Automate daily, weekly and seasonal scaling where operations finish before peak demand, and keep manual scaling for unpredictable spikes and critical events.
4. Monitor data movement costs. Track network use, storage I/O and rebalance duration to tune rebalancing and spot when tiered storage settings need adjusting.
5. Plan manual interventions for high-impact events. Campaigns, launches and other planned events should trigger manual scaling ahead of demand. Auto-scaling gives you efficiency, not emergency capacity.

Kafka auto-scaling takeaways

Tiered storage plus Kubernetes-native operators is a big step toward elastic Kafka, but it’s still early. Proposals for fully disaggregated, “diskless” storage, such as KIP-1150, aim to go further by removing most local storage, so scaling would need almost no data movement and brokers would behave much more like stateless services.

For now, the OSO engineers see auto-scaling as a powerful efficiency tool rather than a complete answer to capacity planning. Teams can save significant cost and operational effort by auto-scaling predictable workloads thoughtfully.

Key takeaway: tiered storage shrinks the local data each broker holds, so rebalances move about a tenth of the data, which makes automated scaling practical for predictable load patterns but not for sudden spikes.

Kafka auto-scaling isn’t about eliminating capacity planning. It automates routine scaling so engineers can focus on the exceptional cases that need human judgment. The elasticity barrier isn’t gone, but it’s much lower. For teams running Kafka on Kubernetes with predictable traffic, auto-scaling now offers a practical route to better resource efficiency without giving up production reliability.

Related reading:

• Best managed Kafka services
• Building bulletproof disaster recovery for Apache Kafka

Frequently asked questions