blog by OSO

How Kafka Consumer 4.0’s New Rebalance Protocol Eliminates the Two-Phase Bottleneck

Sion Smith
Sion Smith 13 June 2025
OSO guide to the Kafka 4.0 consumer rebalance protocol (KIP-848), explaining server-side assignment and incremental rebalancing, with white title text on a navy panel and a lime Blogs label.

Updated October 2026: We checked this guide against the current Apache Kafka documentation. The new protocol is enabled on brokers by default since 4.0 but still has to be switched on in the client. Under KIP-1274, Kafka 5.0 makes it the KafkaConsumer default and Kafka 6.0 makes it the only protocol the Java consumer supports, while brokers keep supporting classic groups. We also removed benchmark figures we could not reproduce from public sources.

In distributed streaming, few things are more frustrating than watching a whole consumer group grind to a halt because one slow member can’t keep up during a rebalance. OSO engineers have seen this many times in enterprise deployments: a single unresponsive consumer holding dozens of healthy instances hostage, with delays rippling through real-time pipelines.

The new consumer rebalance protocol is the biggest change to consumer group management since Kafka’s early days. It replaces the client-driven, synchronous rebalance with incremental, server-side reconciliation, removing the group-wide synchronisation barrier that has hurt large deployments for years.

Kafka consumer rebalance protocol

Why the classic protocol stalls the whole group

The classic protocol is built around three RPCs: JoinGroup, SyncGroup and Heartbeat. When membership or subscriptions change, every member must rejoin, the coordinator waits for all of them, one consumer is elected leader and computes the assignment client-side, and SyncGroup distributes the result. Nobody moves on until everybody has arrived.

With the eager assignors (Range, RoundRobin, Sticky), every member revokes all of its partitions at the start of each rebalance. CooperativeStickyAssignor improved this by letting members keep partitions that are not moving, but it still needs the group-wide barrier, and moving a partition takes two rounds. Here is what happens when consumer C joins a group with the cooperative assignor:

// Classic protocol with CooperativeStickyAssignor
Consumer A: owns partitions [1, 3]
Consumer B: owns partitions [2]
Consumer C: joining group

// Rebalance 1
1. C sends JoinGroup
2. A and B learn about the rebalance from their heartbeat responses
3. A and B send JoinGroup with their current assignments
4. The coordinator waits for ALL members before proceeding
5. The leader (A) computes the new assignment
6. Partition 3 is removed from A but not yet given to C

// Rebalance 2
7. A finishes revoking partition 3 and triggers another rebalance
8. All members rejoin again
9. The leader assigns partition 3 to C

Every barrier is a chance for one slow member to delay everyone. A long garbage collection pause, a slow onPartitionsRevoked callback or a consumer stuck in a long poll() loop holds up the whole group, and OSO engineers have watched rebalances stretch to tens of seconds because of a single struggling instance. The bigger the group, the more likely it is that at least one member is slow at the moment a rebalance starts.

A typical classic configuration looks like this:

group.protocol=classic
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
session.timeout.ms=45000
heartbeat.interval.ms=3000
max.poll.interval.ms=300000
enable.auto.commit=true
auto.commit.interval.ms=5000

While a classic rebalance is in progress, commits can fail with RebalanceInProgressException or CommitFailedException, so any work since the last successful commit may be reprocessed.

Kafka Consumer 4.0 New Rebalance Protocol

How the new protocol moves assignment to the broker

With the new protocol, the group coordinator on the broker owns the assignment. Each member simply tells the coordinator what it subscribes to, and the coordinator runs a server-side assignor, uniform by default or range, to compute a target assignment. Each member then reconciles towards its target on its own schedule.

The same scenario now looks like this:

// New consumer protocol: one incremental rebalance
Consumer A: owns partitions [1, 3]
Consumer B: owns partitions [2]
Consumer C: joining group

1. C starts sending ConsumerGroupHeartbeat requests to join
2. The coordinator computes a new target assignment (partition 3: A -> C)
3. A's next heartbeat response tells it to revoke partition 3
4. A revokes partition 3 and confirms on its next heartbeat
5. C's next heartbeat response assigns partition 3
6. Done. B was never interrupted and there was no group-wide barrier

A partition is only handed to its new owner after the previous owner has confirmed the revocation, so ownership never overlaps. Members that are not affected by a change keep fetching and committing as normal. A slow member only delays the partitions it is giving up, and if it misses its session timeout the coordinator removes it and reassigns its partitions.

The client-side configuration gets simpler too:

group.protocol=consumer
group.remote.assignor=uniform
max.poll.interval.ms=300000

One heartbeat RPC instead of three

The new protocol folds joining, assignment and liveness into one RPC, ConsumerGroupHeartbeat. Each heartbeat carries the member’s epoch, subscription and owned partitions, and each response can carry a new assignment. Assignment changes are delivered incrementally, so there is no separate join or sync phase and no client leader.

That has a few practical effects:

  • No stop-the-world rebalances. Membership changes only affect the partitions that move.
  • Simpler clients. The assignment logic lives on the broker, so every client language gets the same behaviour.
  • Server-side regex subscriptions. subscribe(SubscriptionPattern) evaluates RE2J patterns on the broker rather than in each client.
  • A new threading model. The Java consumer that implements the protocol runs network and coordinator work on a background thread, with new consumer metrics that expose it.

Your rebalance listener still works. onPartitionsRevoked and onPartitionsAssigned are called with only the partitions that are actually moving:

consumer.subscribe(List.of("orders", "payments"), new ConsumerRebalanceListener() {
    @Override
    public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
        // Only the partitions moving away from this member
        consumer.commitSync(currentOffsets(partitions));
    }

    @Override
    public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
        // Only the partitions newly given to this member
        log.info("Assigned {}", partitions);
    }
});

while (running) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
    for (ConsumerRecord<String, String> record : records) {
        process(record);
    }
    // Members keep committing for the partitions they still own
    consumer.commitAsync();
}

Classic vs consumer protocol at a glance

AspectClassic protocolConsumer protocol (KIP-848)
StatusSupported; default until Kafka 5.0GA since Kafka 4.0; client opt-in until 5.0
Who computes assignmentsElected client leaderGroup coordinator on the broker
RPCsJoinGroup, SyncGroup, HeartbeatConsumerGroupHeartbeat
Group-wide barrierYes, every rebalanceNo, members reconcile independently
Assignorspartition.assignment.strategy on the clientuniform or range on the broker (group.remote.assignor to choose)
Heartbeat and session timeoutClient heartbeat.interval.ms and session.timeout.msBroker group.consumer.* defaults, overridable per group
Regex subscriptionsEvaluated by each clientRE2J evaluated on the broker
Custom client-side assignorsSupportedNot supported

Configuration changes when you switch

When group.protocol=consumer is set, these client settings and APIs are no longer usable:

  • heartbeat.interval.ms
  • session.timeout.ms
  • partition.assignment.strategy
  • enforceRebalance()

Heartbeat and session timeouts move to the broker (group.consumer.heartbeat.interval.ms and group.consumer.session.timeout.ms) and can be overridden per group with consumer.heartbeat.interval.ms (default 5000) and consumer.session.timeout.ms (default 45000). Pick the server-side equivalent of your old assignor:

Client-side assignorServer-side assignor
RangeAssignorrange
CooperativeStickyAssignoruniform
StickyAssignoruniform
RoundRobinAssignoruniform
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "orders-service");
props.put(ConsumerConfig.GROUP_PROTOCOL_CONFIG, "consumer");
// Optional: choose the server-side assignor ("uniform" or "range")
props.put(ConsumerConfig.GROUP_REMOTE_ASSIGNOR_CONFIG, "uniform");
// Remove heartbeat.interval.ms, session.timeout.ms and
// partition.assignment.strategy: they are rejected with group.protocol=consumer

For a full list of consumer settings worth reviewing at the same time, see our Kafka consumer configuration cheatsheet.

Migrating a live consumer group

The brokers must run Kafka 4.0 or later, where the protocol is enabled by default through the group.version feature. After that you have two options:

  1. Online (no downtime). Roll out consumers with group.protocol=consumer one instance at a time. When the first new-protocol member joins, the coordinator converts the group from classic to consumer and interoperates with the remaining classic members until the rollout finishes. This only works if the classic group uses an assignor that does not embed custom metadata.
  2. Offline. Stop every consumer in the group, then start them again with the new setting. Empty groups are converted automatically.

Downgrading works the other way round: the group converts back to classic when the last new-protocol member leaves.

Before you roll out, check for custom client-side assignors, rack-aware assignment and code that calls enforceRebalance(), and update any per-consumer timeout tuning to the broker or per-group configs. Our deep dive on how KIP-848 works under the hood covers the reconciliation process in more detail.

Monitoring and limitations to plan for

Rebalances under the new protocol are easier to reason about because each one has explicit member epochs and only touches the partitions that move. The Java consumer also adds new metrics for its background thread, so you can see whether delays come from the network, the coordinator or your own processing. Keep watching consumer lag and max.poll.interval.ms violations: a consumer that stops polling is still removed from the group.

There are two current limitations, according to the Apache Kafka documentation:

  • Client-side assignors are not supported. Custom assignment logic has to move to the broker as a ConsumerGroupPartitionAssignor implementation or be dropped.
  • Rack-aware assignment is not fully supported yet. Work is tracked in KAFKA-19387.

What comes next: Kafka 5.0 and 6.0

KIP-1274 sets out the timeline for the protocol:

Kafka versionConsumer protocol status
3.7Early access
4.0Generally available; enabled on brokers, opt-in on clients
5.0 (planned)KafkaConsumer defaults to the consumer protocol, classic still supported
6.0 (planned)KafkaConsumer only supports the consumer protocol; brokers still support classic groups

Key takeaway: Set group.protocol=consumer on Kafka 4.0+ to remove the group-wide rebalance barrier, after replacing client-side assignor and timeout settings.

The synchronisation barrier was the main reason large consumer groups were fragile. Removing it lets groups scale out, absorb slow members and deploy rolling updates without stop-the-world pauses. With the new protocol becoming the default in Kafka 5.0, now is a good time to test it on a non-critical group and plan the rest of your migration.

Frequently asked questions

What is the new Kafka consumer rebalance protocol?

It is the next-generation consumer group protocol from KIP-848, generally available since Apache Kafka 4.0. The broker’s group coordinator computes partition assignments with a server-side assignor and delivers changes incrementally through the ConsumerGroupHeartbeat RPC, so there is no group-wide JoinGroup/SyncGroup barrier.

How do I enable the new consumer protocol?

Run Kafka 4.0 or later brokers, where it is enabled by default, and set group.protocol=consumer in your Kafka 4.0+ consumer configuration. Optionally set group.remote.assignor to uniform or range, and remove heartbeat.interval.ms, session.timeout.ms and partition.assignment.strategy from the client.

Can I migrate a consumer group without downtime?

Yes. Roll out consumers with group.protocol=consumer one at a time. When the first new-protocol member joins, the group converts from classic to consumer and interoperates with classic members until the rollout completes. This only works if the classic group’s assignor does not embed custom metadata.

Is the classic consumer protocol being removed?

Not from brokers. Under KIP-1274, Kafka 5.0 makes the consumer protocol the KafkaConsumer default while still supporting classic, and Kafka 6.0 makes the Java KafkaConsumer support only the consumer protocol. Brokers keep supporting classic groups for backward compatibility.

What are the limitations of the new protocol?

Custom client-side assignors are not supported, so custom logic must move to a server-side ConsumerGroupPartitionAssignor. Rack-aware assignment is not fully supported yet, and the client settings heartbeat.interval.ms, session.timeout.ms, partition.assignment.strategy and enforceRebalance() cannot be used.

Looking to upgrade to Kafka 4.0

Have a conversation with one of our experts to discover how we support your team through this upgrade.

CONTACT US
OSO
Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.