blog by OSO

Spring Kafka: Produce, Consume and Handle Errors

Sion Smith
Sion Smith 17 November 2026
OSO guide to Spring Kafka producers, listeners and error handling, with white title text on a navy panel and a lime Blogs label.

What is Spring for Apache Kafka?

Spring for Apache Kafka (the spring-kafka library) applies core Spring ideas to Kafka. It wraps the Apache Kafka Java client in a template for sending and in message-driven POJOs for receiving.

In a Spring Boot application you add the Kafka starter and set a few spring.kafka.* properties. Boot then auto-configures a KafkaTemplate, a consumer factory and a listener container factory. Take the current library and Boot versions from the Spring documentation and start.spring.io rather than copying them from old tutorials.

Three pieces do most of the work:

  • KafkaTemplate sends records to a topic and returns a future you can inspect.
  • @KafkaListener turns a method into a consumer. A listener container manages polling, threads and offset commits for you.
  • Error handlers and transaction managers decide what happens when processing fails.

Spring does not change what Kafka guarantees. Delivery semantics still come from your Kafka configuration, so tune the client settings in Kafka producer settings first.

How do you produce and consume messages?

Producing is a single call on the template. The key controls the partition, so pick one that keeps related events together.

@Service
public class OrderPublisher {

    private final KafkaTemplate<String, String> template;

    public OrderPublisher(KafkaTemplate<String, String> template) {
        this.template = template;
    }

    public void publish(String orderId, String json) {
        template.send("orders", orderId, json);
    }
}

Consuming is just as short. The listener runs on a container thread, and the container commits offsets after your method returns.

@Component
public class OrderListener {

    @KafkaListener(topics = "orders", groupId = "order-service")
    public void onOrder(String json) {
        // process one record
    }
}

Set spring.kafka.listener.concurrency to run several consumer threads in one application instance. Keep it at or below the partition count, because extra threads sit idle. The Kafka consumer configuration cheatsheet covers the client settings behind these defaults.

SettingWhere you set itWhat it controls
spring.kafka.consumer.group-idBoot propertyDefault consumer group for listeners
groupId on @KafkaListenerAnnotationOverrides the group for one listener
spring.kafka.listener.concurrencyBoot propertyConsumer threads per listener
spring.kafka.listener.ack-modeBoot propertyWhen offsets are committed

How do you handle errors and retries?

Listener errors are handled by the container’s error handler. In current Spring Kafka that is DefaultErrorHandler, which replaced SeekToCurrentErrorHandler in version 2.8.

Out of the box it retries a failed record with no delay for a small number of attempts, then logs and moves on. That is rarely what you want in production. Give it a back-off and a recoverer that publishes the poison record to a dead-letter topic.

@Bean
public DefaultErrorHandler errorHandler(KafkaTemplate<String, String> template) {
    DeadLetterPublishingRecoverer recoverer =
        new DeadLetterPublishingRecoverer(template);
    return new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 3L));
}

Blocking retries hold up the partition while they wait. If a downstream outage could last minutes, use non-blocking retries with @RetryableTopic, which moves failed records through retry topics and then to a dead-letter topic. It does not support batch listeners, so batch consumers should use DefaultErrorHandler with a recoverer.

Check that the dead-letter topic exists and has at least as many partitions as the source topic, because the default resolver keeps the original partition number. Our guide to Kafka error handling tips covers the wider patterns.

Key takeaway: Treat Spring Kafka as a thin, well-configured client: get error handling, commits and concurrency right before you add more framework features.

How do transactions and batch listeners work?

Set spring.kafka.producer.transaction-id-prefix and Spring Boot creates a KafkaTransactionManager for you. The listener container then starts a Kafka transaction before it calls your listener.

Any KafkaTemplate send inside the listener joins that transaction. When the method succeeds, the consumed offsets go into the transaction and everything commits together. If it fails, the transaction rolls back and the record is redelivered.

This gives you read-process-write between Kafka topics. It does not cover side effects such as database writes or HTTP calls, and consumers must read with isolation.level=read_committed to see only committed results. Each application instance needs a unique transaction id prefix.

Transactions add latency and operational care, so use them only when duplicate output records are a real problem.

Batch listeners receive a list of records from one poll.

@KafkaListener(topics = "orders", containerFactory = "batchFactory")
public void onBatch(List<String> messages) {
    // process the whole poll in one go
}

You enable them with setBatchListener(true) on the container factory. The default ack mode, BATCH, commits offsets after the whole poll has been processed. Batches suit bulk database writes, but one bad record can fail the whole poll unless your error handler can isolate it.

What should you configure for production?

Most teams change a short list of properties before going live.

AreaProperty or settingWhy it matters
Connectionspring.kafka.bootstrap-serversPoint at several brokers, not one
Securitysecurity.protocol, SASL and SSL propertiesEncrypt traffic and authenticate clients
Throughputspring.kafka.listener.concurrencyMatch threads to partitions
Commitsspring.kafka.listener.ack-modeChoose when offsets are committed
HealthidleEventInterval on the containerDetect listeners that stop receiving records
Rebalancinggroup.protocolOpt in to the new rebalance protocol

Spring Kafka 4.0 supports the new consumer rebalance protocol from KIP-848. You opt in with spring.kafka.consumer.properties.group.protocol=consumer, and the documented default stays classic. Read the new consumer rebalance protocol before switching, and test it on a non-production cluster first.

Wire the Micrometer metrics into your monitoring, and alert on consumer lag as well as listener errors. If your Spring services depend on a cluster you do not fully trust yet, our Kafka consulting services and Kafka support teams can review the configuration with you.

Frequently asked questions

What is the difference between Spring Kafka and the plain Java client?

Spring Kafka wraps the Apache Kafka Java client. It adds KafkaTemplate, listener containers, error handlers and Spring Boot configuration, so you write less polling and commit code yourself.

How do you create a @KafkaListener in Spring Boot?

Add the Kafka starter, then annotate a method in a Spring bean with @KafkaListener(topics = "orders", groupId = "order-service"). Boot configures the container factory and starts the listener with the application.

What replaced SeekToCurrentErrorHandler?

DefaultErrorHandler replaced it in Spring Kafka 2.8. It combines retries, a configurable back-off and an optional recoverer, such as one that publishes failed records to a dead-letter topic.

Does Spring Kafka support exactly-once transactions?

Yes, for Kafka-to-Kafka processing. Set a transaction id prefix and the container commits produced records and consumed offsets together. External side effects such as database writes are not covered.

How do you set the consumer group id in Spring Kafka?

Set spring.kafka.consumer.group-id for the default, or groupId on the @KafkaListener annotation to override it for one listener. Listeners sharing a group id share the partitions.

Can Spring Kafka consume records in batches?

Yes. Enable batch mode on the listener container factory and declare the listener parameter as a list. The container then passes each poll’s records to your method in one call.

Stuck on a Spring Kafka consumer?

OSO engineers run Apache Kafka in production for banks, retailers and the public sector. Tell us what you are weighing up and we will give you a straight answer.

CONTACT US
OSO
Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.