Kafka Streams Rebalances and Assignments 101

Updated October 2026: We checked this guide against the current Apache Kafka documentation. We corrected the consumer configuration names (max.poll.interval.ms, heartbeat.interval.ms, session.timeout.ms), replaced a link to the Kafka 2.1 docs with the current Streams configuration reference, and added the warm-up replica settings and the new Streams rebalance protocol (KIP-1071), which is production-ready from Kafka 4.2.

OSO has run plenty of Kafka Streams applications in production, and rebalances and task assignments are part and parcel of that architecture. This guide explains why Kafka Streams rebalances, what happens during a rebalance, and how to tune and monitor rebalances so they stay short and rare.

Why does Kafka Streams rebalance?

A rebalance is how a Kafka Streams application redistributes tasks (one task per input partition group) across its running instances. It is triggered by a specific event or condition:

Probing rebalances come from the high-availability task assignor introduced by KIP-441. When a stateful task should move to an instance (call it B) whose local state is behind, Streams keeps the task active where it is and assigns B a warm-up replica instead. B restores state in the background, and a probing rebalance checks whether B is within acceptable.recovery.lag (10,000 offsets by default). Once it is, the task is handed over. A probing rebalance often changes little or nothing, but it keeps the assignment converging towards a balanced state without long restoration pauses.

What happens during a Kafka Streams rebalance?

During a rebalance, tasks may change ownership from one instance to another. With the old eager protocol, every instance revoked all of its tasks and processing stopped until the rebalance finished. Since Kafka 2.4, Kafka Streams uses incremental cooperative rebalancing, so instances keep processing the tasks they retain and only the tasks that actually move are paused.

When a task moves, the instance giving it up flushes its state stores and buffers and commits the task’s offsets. That triggers a follow-up rebalance, after which the new owner starts processing the task. This two-step handoff keeps the transition smooth and the downtime per task small.

A single change can therefore involve several rebalances, depending on how many tasks move and how long instances take to close them out. When the group produces a stable assignment with no follow-up rebalance scheduled, the rebalance is complete and the application has converged.

From Kafka 4.2, you can also opt into the new Streams rebalance protocol (KIP-1071) with group.protocol=streams. The broker’s group coordinator computes task assignments instead of a client-side leader, and changes are handed out incrementally through heartbeats. It needs broker-side enablement, supports the sticky assignor only, and existing applications migrate offline rather than with a rolling bounce, so test it before switching production workloads.

How do you reduce Kafka Streams rebalances?

Rebalances are necessary, but unnecessary ones waste processing time. Start with the consumer settings that decide when an instance is considered dead:

Larger timeouts make the group less sensitive to slow or paused instances, which prevents needless rebalances. The trade-off is that a genuinely crashed instance is detected later, so its tasks stay unassigned for longer.

When many stateful tasks need to migrate, for example after scaling out, the warm-up and probing phase decides how long the application takes to rebalance fully:

max.warmup.replicas=2                  # default; raise to warm up more tasks at once
probing.rebalance.interval.ms=600000   # default 10 minutes; minimum 1 minute
acceptable.recovery.lag=10000          # offsets behind before a task counts as caught up
num.standby.replicas=1                 # default 0; standbys make failover near-instant

Increasing max.warmup.replicas lets more tasks restore in parallel, at the cost of extra broker traffic. The maximum migration rate is roughly max.warmup.replicas divided by probing.rebalance.interval.ms. Standby replicas keep a shadow copy of state on another instance, so a failed task can restart with little or no restoration. For more on how state stores behave, see our guide to custom Kafka Streams state stores.

Which metrics show Kafka Streams rebalance problems?

To understand what is happening during a rebalance, monitor the embedded consumer metrics. The full list of settings is in the Kafka Streams configuration reference, and the consumer settings are in the consumer configuration docs.

If last-rebalance-seconds-ago keeps resetting at roughly the probing interval, warm-up replicas are still catching up. If it resets at random times, look for poll or heartbeat timeouts. Our Kafka performance monitoring best practices cover alerting on these signals.

What are the key lessons for Kafka Streams rebalances?

Rebalances are an essential part of keeping a Kafka Streams application balanced and healthy. Understanding why they happen, tuning the settings that trigger them and monitoring the right metrics makes them predictable rather than disruptive.

Rebalances are not something to fear. They are the mechanism that keeps work evenly spread and state available. If you are also running plain consumers, read how the new consumer rebalance protocol in Kafka 4.0 removes the group-wide barrier.

Key takeaway: Most Kafka Streams rebalances are either expected (deploys, scaling, probing rebalances for warm-up replicas) or caused by missed poll and heartbeat deadlines. Tune max.poll.interval.ms and session.timeout.ms, use static membership and standby replicas, and alert on rebalance metrics so the cooperative protocol can do its job quietly.

Frequently asked questions