Kafka queues in Apache Kafka 4.0 via Share Groups

For years, Apache Kafka’s consumer group mechanism has been the backbone of how applications consume data from Kafka topics. The model is powerful and well suited to many use cases, but it imposes constraints that have led engineers to build workarounds. What if you need to scale your consumers beyond your partition count? What if your messages need very different processing times? What if your workload has sharp peaks and troughs that demand elastic scaling?

Apache Kafka 4.0 introduced a feature that addresses these challenges: “Queues for Kafka”, delivered as share groups. The name is intentionally provocative. It acknowledges that many organisations have been trying to use Kafka as a queue, despite its design as a distributed log.

Share groups remove a key limitation of the consumer group model by letting multiple consumers cooperatively process records from the same partition. That simple change unlocks patterns previously dominated by traditional message queues. This article explains how share groups work, when to use them and how to get started.

Updated October 2026: share groups shipped as early access in Kafka 4.0, as a preview in 4.1 and became production-ready in Kafka 4.2. The status, configuration names, API and tools below have been checked against the Apache Kafka 4.3 documentation and Javadoc. The 4.0 early-access setup and API names in the original version of this article no longer apply.

What are the limitations of traditional consumer groups?

To appreciate share groups, you first need to understand the constraints of Kafka’s traditional consumption model.

Consumer groups follow a strict partition-to-consumer coupling. When a consumer joins a group, the group coordinator assigns it specific partitions, and it owns them exclusively within that group. At any time, each partition is processed by only one consumer in the group.

The consequences show up as soon as you need to scale. If your topic has five partitions, you can have at most five active consumers in a group, and any extra consumers sit idle. Parallelism has a hard upper limit tied to the partition count.

This has led many organisations to over-partition: create far more partitions than they need today to allow for future growth. “We need five consumers now, but twenty at peak, so let’s create twenty partitions.” It works, but over-partitioning uses more broker resources, makes rebalancing heavier and can lead to smaller, less efficient batches. Adding partitions later is possible but disruptive when you rely on keys for ordering.

Variable processing time is the second problem. Imagine most messages take milliseconds, but a few take seconds or minutes because of complex calculations or slow external calls. In a consumer group, a consumer cannot move past the current message in a partition until it has finished with it, so one slow message holds up the whole partition while other consumers have nothing to do.

Teams work around this with over-partitioning, intermediate topics or custom per-message commit tracking. Share groups build a better answer into Kafka itself.

How do Kafka share groups work?

Share groups take a different approach. Instead of giving each consumer exclusive ownership of partitions, they let several consumers process records from the same partition at the same time. The number of consumers in a share group can exceed the number of partitions in a topic.

Andrew Schofield, the lead author of KIP-932, describes an important distinction between a topic partition and a share partition. A topic partition is the familiar unit of parallelism. A share partition is the view of a topic partition from one particular share group.

This shifts responsibility to the broker. In consumer groups, consumers control their position in the log by committing offsets. In share groups, the broker decides which records to deliver next, which is how it can spread the records of one partition across several consumers.

Three characteristics stand out:

1. Cooperative consumption. Several share consumers can be assigned the same partition and receive different records from it, which breaks the link between partition count and consumer count.
2. Scaling beyond partitions. A topic with three partitions can have ten or more active share consumers. This helps workloads with large swings in volume or very variable processing times.
3. Per-record acknowledgement. Instead of committing the highest processed offset, a share consumer acknowledges each record: accept it, release it for another attempt, reject it as unprocessable or renew its lock because it is still working on it.

Under the hood, the broker tracks the delivery state of each in-flight record. When a record is fetched by a share consumer it is acquired under a time-limited acquisition lock, 30 seconds by default (share.record.lock.duration.ms), and no other consumer in the group can receive it. The consumer then accepts, releases, rejects or renews it. If it does nothing, for example because it crashed, the lock expires and the record becomes available again for another delivery attempt.

Every delivery attempt is counted. Once a record reaches the delivery count limit, 5 by default, it is archived and not delivered again. A rejected record is archived straight away. The broker also caps how many records can be acquired per share partition at once (2,000 by default in Kafka 4.3). When the cap is reached, fetches return nothing new until records are acknowledged or their locks expire.

Records still live on durable, replicated Kafka topics with normal retention, and the same topic can be read by consumer groups and other share groups independently. Share groups add queue-like consumption on top of Kafka’s log; they do not replace it.

What are the use cases for Kafka share groups?

Share groups make several patterns much easier than they are with consumer groups.

Elastic scaling for unpredictable load. Many systems have sharp traffic peaks: an e-commerce platform during a flash sale, a payment processor at peak shopping hours or an analytics system that receives batch uploads at set times. With consumer groups you over-partition for the peak. With share groups you keep a baseline of consumers and add more to the same share group at peak, without changing the topic.

For example, a topic with 5 partitions normally needs 5 consumers. At peak, processing demand triples. With consumer groups you would create 15 partitions up front and leave two thirds of them underused most of the time. With a share group you keep 5 partitions and scale from 5 to 15 consumers at peak, then back down.

Work queues for long-running tasks. When each message is an independent task that may take seconds or minutes, such as image processing, complex calculations or workflows calling several external services, a consumer group blocks a partition behind every slow task. A share group lets many workers pull the next available task. A financial services firm distributing risk calculations of very different complexity, for instance, can run a pool of workers that always pick up the next calculation.

Handling poison messages. A malformed message or an unexpected edge case can block a partition in a consumer group until someone intervenes. Share groups count delivery attempts and archive a record that keeps failing, so processing carries on. A dead-letter queue for share groups has been proposed in a KIP but is not part of a release as of Kafka 4.3, so if you need to keep failed records, write them to your own failure topic before rejecting them.

These use cases mean workloads that once needed a separate queueing system alongside Kafka can often run on Kafka alone, which simplifies the architecture.

How do you get started with Kafka share groups?

Use Kafka 4.2 or later for anything beyond experiments. The 4.0 early access is not compatible with later versions and should not be used.

On a new Kafka 4.2 or later cluster, share groups are enabled by default. On a cluster upgraded from an earlier version, enable the feature and check it:

bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --feature share.version=1
bin/kafka-features.sh --bootstrap-server localhost:9092 describe

Share groups store their state in a new internal topic, __share_group_state. Like other internal topics it uses 3 replicas by default, so on a cluster with fewer than 3 brokers set the broker configs for the share coordinator state topic replication factor and min ISR to 1 before first use.

A share consumer looks very like a normal consumer. You create a KafkaShareConsumer instead of a KafkaConsumer, and every consumer with the same group.id joins the same share group.

Share consumer with implicit acknowledgement

With the default implicit acknowledgement mode, every record returned by poll is treated as successfully processed when you next call poll, commitSync or commitAsync:

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaShareConsumer;

import java.time.Duration;
import java.util.List;
import java.util.Properties;

public class BasicShareConsumer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("group.id", "my-share-group");
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

        try (KafkaShareConsumer<String, String> consumer = new KafkaShareConsumer<>(props)) {
            consumer.subscribe(List.of("my-topic"));
            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
                for (ConsumerRecord<String, String> record : records) {
                    process(record);
                }
                // The next poll() implicitly accepts every record returned by this one.
            }
        }
    }

    private static void process(ConsumerRecord<String, String> record) {
        System.out.printf("partition=%d offset=%d value=%s%n", record.partition(), record.offset(), record.value());
    }
}

Share consumer with explicit acknowledgement

For production workloads you usually want explicit acknowledgement. Set share.acknowledgement.mode to explicit and acknowledge every record with an AcknowledgeType before the next poll. If you call poll with records still unacknowledged, the consumer throws an IllegalStateException.

import org.apache.kafka.clients.consumer.AcknowledgeType;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaShareConsumer;

import java.time.Duration;
import java.util.List;
import java.util.Properties;

public class ExplicitAckShareConsumer {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        props.put("group.id", "orders-share-group");
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("share.acknowledgement.mode", "explicit");

        try (KafkaShareConsumer<String, String> consumer = new KafkaShareConsumer<>(props)) {
            consumer.subscribe(List.of("orders"));
            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
                for (ConsumerRecord<String, String> record : records) {
                    try {
                        processOrder(record);
                        consumer.acknowledge(record, AcknowledgeType.ACCEPT);
                    } catch (TransientException e) {
                        consumer.acknowledge(record, AcknowledgeType.RELEASE); // deliver again later
                    } catch (Exception e) {
                        consumer.acknowledge(record, AcknowledgeType.REJECT);  // archive, no more attempts
                    }
                }
                consumer.commitSync(); // commit the acknowledgements now
            }
        }
    }

    static void processOrder(ConsumerRecord<String, String> record) throws TransientException {
        // business logic
    }

    static class TransientException extends Exception {
    }
}

The acknowledgement types are:

Work queue for long-running tasks

For tasks that take longer than the lock duration, take one record at a time and renew the lock while you work. Setting share.acquire.mode to record_limit makes the consumer return no more than max.poll.records records per poll and stops it prefetching, so locks start only when you are ready:

props.put("share.acknowledgement.mode", "explicit");
props.put("share.acquire.mode", "record_limit");
props.put("max.poll.records", "1");

During a long task, call consumer.acknowledge(task, AcknowledgeType.RENEW) from the polling thread before the lock expires to extend it, then ACCEPT or REJECT the task when it finishes. Alternatively, raise the group’s lock duration so it covers your longest task.

Share group configuration

Most share group behaviour is set per group with kafka-configs.sh --entity-type groups, within limits set by broker configs. These are not consumer properties, so setting them in the consumer’s Properties has no effect.

For example, to give a share group the broker’s default maximum of 1 minute per delivery:

bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter \
  --entity-type groups --entity-name work-queue-share-group \
  --add-config share.record.lock.duration.ms=60000

The lock duration must stay within the broker’s minimum and maximum (by default 15 seconds to 1 minute in Kafka 4.3), so for longer tasks either raise the broker maximum or renew the lock with RENEW.

Share group tools

Kafka ships command-line tools for share groups:

To list share groups and see the start offset and lag of one:

bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --list
bin/kafka-share-groups.sh --bootstrap-server localhost:9092 --describe --group my-share-group

The start offset is the earliest offset of the in-flight records being considered for delivery; some later records may already have been delivered and acknowledged. Add --members to see which consumers are in the group and which partitions each is assigned.

The most common pitfall is acknowledgement. In explicit mode, forgetting to acknowledge a record makes the next poll throw. In either mode, a record that is never acknowledged is redelivered when its lock expires until it hits the delivery limit, so long-running work must renew its lock or finish within it.

Should you adopt share groups in your architecture?

Share groups are one of the biggest changes to Kafka’s consumption model since consumer groups. By breaking the one-partition-one-consumer rule, they take Kafka into territory previously served by separate messaging systems.

The main benefits are:

• Elastic scaling. Consumer count can change independently of partition count, so you can scale up for peaks and down when it is quiet, which saves money where you pay for compute by usage.
• Resilience. Delivery attempts are counted and failing records are archived, so one malformed message does not block a partition.
• Simpler architecture. Queue-like patterns run directly on Kafka, so teams that ran a separate queue alongside Kafka may be able to consolidate.

Before you adopt them, ask:

• Do your workloads vary a lot in volume or processing time? Share groups handle elastic scaling well.
• Are your messages independent, or do they need strict ordering? Share groups do not preserve ordering across consumers, so keep consumer groups for ordered event streams.
• Are you on Kafka 4.2 or later, and do your client libraries support share groups? The Java client does; check other clients before you commit.

Good first candidates are:

• Background processing tasks with variable completion times
• Systems with sharp peaks and troughs in workload
• Workloads where messages sometimes need a retry or special handling

When you build a proof of concept, pay most attention to acknowledgement and error handling, because they differ most from consumer groups.

Key takeaway: use share groups on Kafka 4.2 or later when records are independent work items that need more consumers than partitions or per-record retries, and keep consumer groups wherever ordering matters.

Share groups combine Kafka’s durable log with queue-like consumption, giving architects a new option for resilient, scalable processing without a second messaging platform.

Frequently asked questions