In the Kafka ecosystem, the Producer API on the client side plays a crucial role in how messages are sent to Kafka brokers. We have lots of experience explaining its components, configurations and how you can optimise it for better performance and reliability. This knowledge is not just theoretical; it is based on real-world support scenarios and extensive testing in over 30 enterprise engagements.
Three internal components shape how a producer behaves, and each one has settings worth understanding before you tune anything.
The record accumulator groups records per partition into batches. Two settings decide when a batch is sent:
batch.size is not held back: it is sent in a batch of its own.linger.ms too high, you will introduce latency when throughput is low.Default change in Kafka 4.0: older guides, including the first version of this post, say linger.ms defaults to 0. Apache Kafka 4.0 changed the default to 5 ms under KIP-1030, because larger batches usually give similar or lower latency overall (see the Kafka 4.0 producer configuration docs). Set linger.ms=0 explicitly if you depend on the old behaviour.
The send buffer is configured with two settings that need to be tuned together:
send() will block when the buffer is full or metadata is unavailable before it throws an exception (60 seconds by default). If you have increased the buffer size, messages can stay in the buffer for longer and you might start seeing exceptions because records have expired, so you will need to revisit this value (and delivery.timeout.ms) if you change the buffer size.Enabling compression can reduce the size of the messages sent over the network and stored on disk, potentially lowering costs. However, the choice of compression algorithm can impact both performance and CPU usage.
| Codec | Typical trade-off |
|---|---|
| none | No CPU cost, largest network and disk footprint |
| gzip | High compression ratio, highest CPU cost |
| snappy | Fast with moderate compression |
| lz4 | Very fast with good compression; a common default choice |
| zstd | Strong compression at moderate CPU cost |
Compression works on whole batches, so it pays off most when batch.size and linger.ms let real batches form.
The acks configuration determines the level of assurance the producer requires before considering a message sent.
| Setting | What the producer waits for | When to use it |
|---|---|---|
| acks=0 | Nothing | Metrics or logs where losing some data is acceptable |
| acks=1 | The partition leader only | Lower latency, with a risk of loss if the leader fails before replicating |
| acks=all | All in-sync replicas | Data you cannot lose (the default since Kafka 3.0, together with idempotence) |
acks=all ensures high reliability because it waits for all in-sync replicas to receive the message, whereas acks=1 or acks=0 can improve performance at the cost of reliability. Combine acks=all with a topic min.insync.replicas of 2 or more for real durability.
There are a number of testing tools available to help you find the optimum configuration values for each of the settings above. Change one setting at a time, measure throughput and end-to-end latency under realistic load, and keep the results so that you can explain every non-default value. We would be happy to help, and to show you our methodology for tuning Kafka producers for high performance.
Key takeaway: tune batching first (
batch.sizewithlinger.ms, which defaults to 5 ms from Kafka 4.0), add compression, keepacks=allwith idempotence for important data, and only growbuffer.memoryandmax.block.mswhen measurements show the buffer is the bottleneck.
Since Apache Kafka 4.0 the producer default for linger.ms is 5 ms. Before 4.0 it was 0, which sent records as soon as possible. KIP-1030 made the change because the efficiency of larger batches usually gives similar or lower latency.
Not on its own. buffer.memory (32 MB by default) only controls how much unsent data the producer can hold; it does not change the batch size. A larger buffer increases memory use and the time records can wait, so tune batch.size and linger.ms first.
It is how long send() blocks when the send buffer is full or metadata is not available before throwing an exception. The default is 60 seconds. Revisit it, and delivery.timeout.ms, if you enlarge the buffer.
LZ4 and zstd are good starting points: LZ4 is very fast, zstd compresses harder at moderate CPU cost. gzip compresses well but uses the most CPU, and snappy is a fast middle ground. Measure with your own payloads.
Usually slightly, because the producer waits for every in-sync replica instead of only the leader. In return it avoids data loss when a leader fails. acks=all with idempotence has been the producer default since Kafka 3.0.
Need help tuning Kafka producers for throughput and reliability? Speak with our engineers about benchmarking and configuring your clients.
CONTACT US