5 Kafka Error Handling Tips

When working with Kafka, it is important to handle errors gracefully to maintain data integrity and ensure message delivery. There are many ways to cater for such errors in your application code when using Kafka. We have seen our fair share, and here are our top 5.

Updated October 2026: the Spring examples now use current Spring for Apache Kafka APIs (version 4.x), and the transactional writes and ordering sections have been corrected: a Kafka transaction synchronised with a database transaction is not atomic across both systems.

What are the 5 Kafka error handling tips?

1. Dead letter topic

The dead letter topic (DLT) pattern handles messages that cannot be processed successfully. Messages with invalid payloads, or that keep throwing exceptions, are routed to a separate topic so the consumer can move on instead of blocking the partition. Keep the original key and record failure details, such as the exception and source topic, partition and offset, in headers so the message can be investigated and replayed.

Be aware that sending one message to a DLT while later messages with the same key carry on breaks per-key ordering. If ordering matters, park every subsequent message for that key until the failed one is fixed.

Dead letter queues are commonly used with Kafka Connect. By default (errors.tolerance=none), any error immediately fails the connector task. For sink connectors, set errors.tolerance=all together with errors.deadletterqueue.topic.name so failed records are written to a dead letter queue topic instead, and make sure that topic is monitored in production.

2. Transactional writes

Use transactional writes when an operation must update both Kafka and a database. Kafka transactions make a group of sends, and any consumer offset commits, atomic within Kafka. Spring can synchronise a Kafka transaction with a database transaction, but the two commit one after the other, so a failure between the commits can still leave them inconsistent. When you need a hard guarantee, write the event to an outbox table in the same database transaction and publish it with change data capture: see our guide to the Apache Kafka outbox pattern. The exact code varies by programming language; contact us if you get stuck.

3. Retry topics

With retry topics, a message that fails is published to a separate retry topic and consumed again after a delay, up to a set number of attempts, before it is sent to the dead letter topic. Because the main topic keeps flowing while the message waits, this non-blocking pattern suits transient failures that may resolve themselves after a few attempts, such as a downstream service being briefly unavailable.

4. Ordered delivery

Kafka guarantees order only within a partition, so ordered delivery starts with giving related messages the same key. On the producer side, idempotence (enabled by default since Kafka 3.0) keeps messages in order even when sends are retried. On the consumer side, use blocking retries rather than retry topics for ordered streams, because a retried message must not be overtaken by the ones behind it. This is particularly important in systems where the order of operations is critical to correctness.

5. Stop on error

There are times when an application should stop if an error occurs. An example is a CDC connector streaming financial transactions from a database. When an error occurs, there is no other path the event can take, so the connector stops and someone has to take action. This is the default behaviour of Kafka Connect (errors.tolerance=none); in Spring for Apache Kafka, use the CommonContainerStoppingErrorHandler.

How do you implement Kafka error handling with Spring?

Now, let’s look at how to implement these patterns with Spring for Apache Kafka, which simplifies the integration with Kafka through its abstraction layer.

Transactional writes implementation

In Spring, enable Kafka transactions by setting a transactionIdPrefix on the producer factory (with Spring Boot, spring.kafka.producer.transaction-id-prefix). With a database transaction manager in place, the @Transactional annotation starts a database transaction and the KafkaTemplate synchronises a Kafka transaction with it:

@Service
public class PendingTransferService {

    private final KafkaTemplate<String, Transfer> kafkaTemplate;
    private final TransferRepository transferRepository;

    public PendingTransferService(KafkaTemplate<String, Transfer> kafkaTemplate,
                                  TransferRepository transferRepository) {
        this.kafkaTemplate = kafkaTemplate;
        this.transferRepository = transferRepository;
    }

    @Transactional
    public void processTransfer(Transfer transfer) {
        transferRepository.save(transfer);
        kafkaTemplate.send("transfers", transfer.getId(), transfer);
    }
}

If the method throws before it returns, both the database write and the Kafka send are rolled back. When the method returns, the database transaction commits first, followed by the Kafka transaction. If the Kafka commit then fails, Spring throws an exception to the caller, but the database change is already committed, so the application has to compensate. Consumers should read with isolation.level=read_committed so they never see records from aborted transactions.

Retry topics implementation

Spring for Apache Kafka sets up retry topics and a dead letter topic for you with the @RetryableTopic annotation on a listener:

@Component
public class TransferListener {

    @RetryableTopic(attempts = 5,
        backOff = @BackOff(delay = 1000, multiplier = 2, maxDelay = 10000),
        include = RemoteServiceNotAvailableException.class)
    @KafkaListener(topics = "transfers", groupId = "transfer-service")
    public void processTransfer(Transfer transfer) {
        // Code to send the transfer to a remote service
    }

    @DltHandler
    public void handleDlt(Transfer transfer) {
        // Record the failure and alert someone
    }
}

This configuration tries the message up to five times in total if a RemoteServiceNotAvailableException is thrown. The delay between attempts starts at one second and doubles each time, up to 10 seconds, and every retry happens on a separate retry topic so the main topic keeps flowing. After the last attempt, the message goes to the dead letter topic and the @DltHandler method is called.

For blocking retries in the listener thread instead, configure a DefaultErrorHandler with a back-off and a DeadLetterPublishingRecoverer on the listener container factory. By default, the DefaultErrorHandler tries a failed record up to 10 times with no delay before logging and skipping it.

Handling ordered delivery

Ordering in Kafka comes from keys and partitions rather than from sending messages one at a time:

@Service
public class OrderMaintainingService {

    private final KafkaTemplate<String, AccountEvent> kafkaTemplate;

    public OrderMaintainingService(KafkaTemplate<String, AccountEvent> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void sendOrderedEvents(List<AccountEvent> events) {
        for (AccountEvent event : events) {
            // Same key -> same partition -> consumed in send order
            kafkaTemplate.send("account-events", event.getAccountId(), event);
        }
    }
}

Every event for one account uses the account ID as its key, so it lands on the same partition and is consumed in the order it was sent. send() is asynchronous, and with idempotence enabled (the default) the producer keeps per-partition order even when it retries, as long as no more than 5 requests are in flight per connection (the default). On the consumer side, use blocking retries for this topic so a failed event is not overtaken by later events for the same account.

Which integration patterns help with errors across services?

Beyond basic error handling, integrating Kafka with other systems like databases and web services needs more sophisticated patterns. Here are three.

Transactional outbox pattern

The transactional outbox pattern ensures reliable messaging when an operation spans a database and Kafka. The service writes the message to an outbox table in the database as part of the same transaction as the business change. A separate process, ideally change data capture with Debezium, then reads the table and publishes the messages to Kafka, so messages are published only if the transaction commits.

Orchestration

Orchestration uses a central coordinator, often a service itself, that controls the interaction between services. The coordinator knows the complete business transaction flow and instructs each service when to perform its task. If a service reports a failure, the coordinator can trigger compensating transactions to roll back earlier changes.

Choreography

Choreography relies on events that are published and consumed by different services autonomously. Each service performs its part of the transaction and publishes an event on completion, which the next service listens to and acts on. The pattern is decentralised, and each service knows what to do next, including publishing a compensating event when its step fails.

Kafka error handling example: what happens when a money transfer fails?

Let’s consider a practical scenario involving a money transfer between banks, which shows how these patterns combine:

1. Initial transaction: a transfer is initiated from a local bank to a foreign bank.
2. Choreography: the local bank publishes an event stating that the money has been debited. The foreign bank, on receiving this event, attempts to credit the amount.
3. Orchestration: if the foreign bank service is down, the orchestrator retries the transaction based on predefined logic, such as the retry topics pattern. If the retries fail, it triggers a compensating transaction that refunds the local bank account.

This example shows how choreography and orchestration can manage complex transactions across different systems and services while keeping them consistent and reliable. Whether you are new to Kafka or running it at scale, reach out to us and we can advise you on the best approach for your use case.

Key takeaway: decide per topic whether a failed message may be skipped, retried later or must stop processing, then use a dead letter topic, retry topics or blocking retries to match, and use the outbox pattern rather than dual writes when Kafka and a database must agree.

Frequently asked questions