The new consumer rebalance protocol KIP-848

Updated October 2026: This post was written in 2023 against the KIP-848 draft. We have checked it against the current Apache Kafka documentation. We fixed the KIP number typos, the timeout and API names, and the description of the range assignor. We also removed client-side delegated assignment, which never shipped, and added the release timeline, the server and client settings, migration steps and current limitations.

Kafka’s consumer group membership and rebalancing protocol served Kafka well for more than a decade, but every rebalance needed every member to stop at a global synchronisation barrier. KIP-848 redesigns it. This article explains what the new consumer rebalance protocol changes, how reconciliation works and what you need to know to adopt it.

What does KIP-848 change in the consumer rebalance protocol?

In the classic protocol, consumers run a JoinGroup and SyncGroup round, and the group leader (one of the consumers) computes the assignment on the client side. A single slow or misbehaving consumer can stall the whole group. KIP-848 changes three things:

How do the session timeout and heartbeat work?

With the new protocol, the broker defines the session timeout and the heartbeat interval. Members send periodic ConsumerGroupHeartbeat requests to the group coordinator to show they are alive. The same request also carries their subscription and the partitions they currently own. The coordinator’s response tells each member its target assignment.

The broker settings are group.consumer.session.timeout.ms (45 seconds by default) and group.consumer.heartbeat.interval.ms (5 seconds by default). A member that does not heartbeat within the session timeout is removed from the group. See the KIP-848 heartbeat API for the full request and response schema.

How are assignments and regex subscriptions handled?

The group coordinator owns the assignment. It tells each member which partitions it should consume at any moment. The server also resolves topic subscriptions, including regular expressions. With the new protocol, consumers subscribe to a pattern with subscribe(SubscriptionPattern). The pattern uses the RE2J format and is evaluated on the server, which keeps user-supplied expressions safe and consistent across members.

What is declarative assignment and reconciliation?

The new protocol replaces client-driven incremental steps with declarative assignment. The coordinator computes the desired target assignment for the whole group, and then reconciles each member towards it. Each member moves through three steps:

1. Revoke partitions that are no longer assigned to it, after committing their offsets.
2. Move to the new member epoch once the coordinator confirms the revocation.
3. Receive new partitions as soon as their previous owners have released them.

The coordinator never assigns a partition to a new owner until the previous owner has revoked it, so two members never consume the same partition at the same time.

Which partition assignors does the new protocol support?

The broker provides two built-in assignors, configured with group.consumer.assignors:

• uniform (the default) spreads partitions as evenly as possible across members.
• range assigns matching partition numbers of co-partitioned topics to the same member, which suits joins across topics with the same partition count.

A consumer can choose one of the configured assignors with group.remote.assignor. Custom assignors run on the broker: you implement the ConsumerGroupPartitionAssignor interface and add the class to group.consumer.assignors. Client-side assignors are not supported. When you migrate, map your old client-side assignor like this:

What happens when a member is slow to revoke partitions?

During reconciliation, partitions often have to move from one member to another. To avoid waiting forever for a member to give up partitions, the coordinator enforces a rebalance timeout. The consumer’s max.poll.interval.ms sets this timeout. If a member does not revoke its partitions in time, the coordinator removes it from the group and the partitions are reassigned.

How does reconciliation work in practice?

Suppose a group has two members, A and B, each with three partitions. When member C joins, the coordinator computes a new target assignment that gives C two partitions: one from A and one from B. C cannot take them straight away, because A and B still own them.

On their next heartbeats, A and B learn their new target assignment. Each commits offsets for the partition it is losing and revokes it. As soon as a partition is released, the coordinator hands it to C. A and B keep processing their other partitions throughout.

Unlike the classic protocol, the coordinator does not wait for the slowest member before assigning anything. If one member is slow or faulty, only the partitions it owns are affected.

How do you upgrade to the new consumer protocol?

The new protocol is enabled on brokers by default from Kafka 4.0, and the group.version feature flag controls it. On the client side, it is opt-in: set group.protocol=consumer. Once you do, the following consumer settings and APIs can no longer be used:

• heartbeat.interval.ms and session.timeout.ms, because the broker now controls them.
• partition.assignment.strategy, because you choose a server-side assignor with group.remote.assignor instead.
• enforceRebalance().

You can migrate an existing group in either of two ways:

• Online: roll out consumers with group.protocol=consumer. When the first one joins, the coordinator converts the group and lets classic and new members work together. This only works if the classic group uses an assignor that does not embed custom metadata.
• Offline: stop all consumers and start them again with the new setting. An empty group is converted automatically.

The release timeline is set out in KIP-1274:

There are known limitations. Client-side assignors are not supported (KAFKA-18327), and rack-aware assignment is still limited (KAFKA-19387). For a hands-on look at the client changes, read our guide to the Kafka consumer 4.0 rebalance protocol.

What are the key takeaways from KIP-848?

KIP-848 moves Kafka consumer group coordination to the broker. Assignments are declarative, reconciliation is incremental and per member, and session and heartbeat settings are managed centrally. Rebalances get faster and less disruptive, a slow consumer no longer stalls the whole group, and consumers get simpler.

For most teams, adopting it means upgrading brokers and clients to Kafka 4.0 or later, choosing the server-side assignor that matches your old client-side one, removing the settings the new protocol ignores and rolling out group.protocol=consumer.

Key takeaway: KIP-848 makes Kafka consumer rebalances incremental and broker-driven. It is generally available from Kafka 4.0 and enabled by default on brokers. Consumers opt in with group.protocol=consumer, and client-side assignors must be mapped to the server-side uniform or range assignors before you migrate.

Frequently asked questions