blog by OSO

Kafka Consumer Configuration Cheatsheet

Sion Smith
Sion Smith 16 January 2025
OSO cheatsheet of key Apache Kafka consumer configuration settings and defaults, with white title text on a navy panel and a lime Blogs label.

Configuring Apache Kafka consumers can be confusing! This cheatsheet explains what each key consumer setting means, gives its default value and shows when to change it. When tuning a consumer, change only one parameter at a time and monitor the effect before moving on.

Updated November 2026: descriptions and default values have been re-checked against the Apache Kafka 4.3 consumer configuration reference. Two corrections: heartbeat.interval.ms should be no higher than one third of session.timeout.ms, and receive.buffer.bytes defaults to 64 KiB (-1 means the operating system default).

Which settings control consumer group membership and rebalances?

These settings decide how the group coordinator detects failed consumers and when a rebalance is triggered.

ConfigurationDefaultDescription
group.instance.idnullA unique, user-provided ID that makes the consumer a static member of its group. If a static member restarts within the session timeout, the group does not rebalance, which reduces the impact on the other consumers.
session.timeout.ms45000 (45 seconds)The timeout used to detect consumer failures. If the broker receives no heartbeat within this time, the consumer is removed from the group and the group rebalances. Must be within the broker’s group.min.session.timeout.ms and group.max.session.timeout.ms.
heartbeat.interval.ms3000 (3 seconds)How often the consumer sends heartbeats. It must be lower than session.timeout.ms and should typically be no higher than one third of it, so several heartbeats can be missed before the session times out.
max.poll.interval.ms300000 (5 minutes)The maximum delay allowed between Consumer#poll() calls. If you don’t call poll() within this interval, the consumer is considered failed and leaves the group, which triggers a rebalance to reassign its partitions.
max.poll.records500How many records a single Consumer#poll() call returns. Make sure you can process all of them within max.poll.interval.ms. It does not change how much data is fetched from the broker.

With the new consumer rebalance protocol (group.protocol=consumer, introduced by KIP-848), session.timeout.ms and heartbeat.interval.ms are not supported on the client. The broker settings group.consumer.session.timeout.ms (default 45 seconds) and group.consumer.heartbeat.interval.ms (default 5 seconds) apply instead. The default group.protocol is still classic.

Which settings control how much data a consumer fetches?

Fetch settings trade latency against throughput and memory use.

ConfigurationDefaultDescription
fetch.min.bytes1The minimum amount of data the broker should return for a fetch request. For example, if set to 1 MB, the broker won’t respond until it has at least 1 MB of data for you or fetch.max.wait.ms passes. Raising it improves throughput at the cost of latency.
fetch.max.wait.ms500The maximum time the broker will wait before answering a fetch request when there isn’t enough data to satisfy fetch.min.bytes.
fetch.max.bytes52428800 (50 MiB)The maximum amount of data the broker will return for a single fetch request. It is not an absolute maximum: if the first record batch is larger, it is still returned so the consumer can make progress.
max.partition.fetch.bytes1048576 (1 MiB)The maximum amount of data the broker will return for a single partition in a fetch request. As with fetch.max.bytes, a larger first batch is still returned.
receive.buffer.bytes65536 (64 KiB)The size of the TCP receive buffer (SO_RCVBUF) used when reading data. Set it to -1 to use the operating system default. Larger buffers can help on high-latency links between consumers and brokers.

Which settings control offset commits and delivery?

These settings decide where a consumer starts reading and how its progress is saved.

ConfigurationDefaultDescription
enable.auto.committrueWhether the consumer commits its offsets automatically in the background. If false, call Consumer#commitSync() or Consumer#commitAsync() explicitly after processing.
auto.commit.interval.ms5000 (5 seconds)How often offsets are committed when enable.auto.commit is true.
auto.offset.resetlatestWhere to start when the group has no committed offset or the offset no longer exists: earliest, latest, by_duration:<ISO 8601 duration> or none (throw an exception).
isolation.levelread_uncommittedHow to handle messages written in transactions (non-transactional messages are always returned). With read_committed, poll() returns only committed transactional messages, up to the last stable offset.

How do you make a consumer fetch from the closest replica?

ConfigurationDefaultDescription
client.rack“” (empty)The rack, data centre or availability zone of the consumer. It corresponds to the broker setting broker.rack. When set, the consumer can fetch from the closest replica, whether it is the leader or a follower.

Follower fetching (KIP-392) only works when each broker has broker.rack set and the brokers use the rack-aware replica selector. By default, brokers return the leader as the preferred read replica. Fetching from a replica in the same availability zone can cut cross-zone network costs in the cloud.

broker.rack=eu-west-2a
replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector

client.rack=eu-west-2a

How should you tune a Kafka consumer?

  • Processing is slow and the consumer keeps leaving the group: lower max.poll.records or raise max.poll.interval.ms so each batch is processed in time.
  • Restarts cause unnecessary rebalances: set group.instance.id for static membership, and make session.timeout.ms longer than a normal restart.
  • High throughput matters more than latency: raise fetch.min.bytes and fetch.max.wait.ms so the broker returns larger batches.
  • You must not lose or skip records: disable auto commit and commit after processing, and choose auto.offset.reset deliberately for new groups.
  • You consume transactional data: set isolation.level=read_committed so aborted messages are not returned.

Key takeaway: keep heartbeat.interval.ms at no more than a third of session.timeout.ms, size max.poll.records so every batch is processed well within max.poll.interval.ms, and change one setting at a time while you watch consumer lag and rebalances.

Frequently asked questions

What is the difference between session.timeout.ms and max.poll.interval.ms?

session.timeout.ms detects a consumer process that has died or lost its connection, because heartbeats are sent from a background thread. max.poll.interval.ms detects a consumer that is alive but stuck, because it limits the time between poll() calls. Exceeding either removes the consumer from the group.

What should heartbeat.interval.ms be set to?

It must be lower than session.timeout.ms and should typically be no higher than one third of it. With the defaults of 3 seconds and 45 seconds, the consumer can miss several heartbeats before it is removed from the group.

What does max.poll.records do?

It limits how many records a single poll() call returns; the default is 500. It does not change how much data is fetched from the broker, because the consumer caches fetched records and hands them out across polls.

When should I use group.instance.id?

Use it when consumers restart frequently, for example during rolling deployments on Kubernetes. A static member that rejoins within the session timeout keeps its partitions, so the rest of the group doesn’t rebalance.

What is the default auto.offset.reset value?

The default is latest, so a new consumer group starts from the end of each partition and skips existing records. Set it to earliest to read from the beginning, or none to fail when no committed offset exists.

Need help with your consumer configuration?

Have a conversation with one of our Kafka experts to discover how we can help.

CONTACT US
OSO
Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.