Configuring Apache Kafka consumers is confusing! This table helps us the remember what each parameter means. When tuning a consumer, consider only changing one parameter at time and monitoring the changes.
| Configuration | Description |
| group.instance.id | The unique member ID of your consumer. This prevents the group from rebalancing when your consumer restarts – reducing the impact on the other consumers. |
| max.poll.records | How many records your Consumer#poll() call will return. Important to ensure you process all of them before max.poll.interval.ms. |
| session.timeout.ms | The timeout used to detect consumer failures. A consumer that doesn’t heartbeat within this time will be marked as failed, kicked out of the group and the consumer group will rebalance. |
| auto.commit.interval.ms | If auto commit is enabled, how often it’s done. |
| fetch.min.bytes | The minimum amount of data the server should return. e.g if set to 1MB, the server won’t return the fetch response until it either has at least 1MB worth of data to give you back, or fetch.max.wait.ms passes. |
| fetch.max.bytes | The maximum amount of data the server will respond with in a single fetch request. |
| isolation.level | How to handle transactional messages from the producer (regular messages are not affected). read_committed means only committed tx messages are returned by poll(). |
| client.rack | The “rack” (or data center, availability zone, etc.) of the client. Setting this allows you to have your consumer fetch from the closest broker (follower or leader). Only works if the broker has replica.selector.class configured correctly. |
| max.poll.interval.ms | The maximum delay you’re allowed to have in between Consumer#poll() calls. If you don’t call poll() within this interval, your consumer will leave the group and cause a rebalance to reassign its partitions to other members. |
| heartbeat.interval.ms | How often your consumer heartbeats. Make sure this is at least 1/3 of session.timeout.ms to allow for 3 possible heartbeats before a potential time out. |
| enable.auto.commit | Whether your consumer will commits its offsets automatically in a background thread. If false, you need to call Consumer#commitSync()/Consumer#commitAsync() explicitly. |
| fetch.max.wait.ms | The maximum amount of time the server will hold up your request in order to wait for the fetch.min.bytes condition. |
| max.partition.fetch.bytes | The maximum amount of data the server will respond with for a single partition in a single fetch request. |
| receive.buffer.bytes | The size of the OS TCP receive buffer (SO_RCVBUF). The default -1 means the OS will tune it itself, but that can be slow for cases where you want to start up reading at full throttle. |
Have a conversation with one of our Kafka experts to discover how we can help.
CONTACT US