Let’s explore the patterns for reprocessing messages in Kafka. When working with Kafka, it is important to handle failures and errors in a robust manner. Spring Boot provides a convenient way to handle these failures, reducing the need for boilerplate code. Even if you are not a Spring Boot expert, you can apply the same principles in your own programming environment. The examples below work with any Kafka-compatible streaming data platform.
What causes Kafka consumers to fail?
We can identify two types of error that can cause consumer order processing to fail: non-transient errors and transient errors.
Error type
Typical causes
How to reprocess
Non-transient
Malformed or wrongly formatted payloads; messages that fail business validation
Route to a dead letter topic with context, fix the cause, then resend
Transient
A temporary failure in an external dependency such as a database or service
Retry with fixed or exponential backoff, then move to a dead letter topic if retries run out
How do you handle non-transient errors in Kafka?
Non-transient errors are not temporary and cannot be recovered automatically. There are two common causes.
Malformed payloads
When serialising and deserialising messages in Kafka, payloads must be properly formatted. If a message payload is malformed or wrongly formatted, it throws an exception during deserialisation. This error happens early in the processing pipeline and is usually caught by utility classes. To handle it, you can use error handling deserialisers in Spring Boot. These deserialisers intercept the message from the source topic and route the failed message to a dead letter topic if deserialisation fails.
For example, in a Spring Kafka consumer you can configure the key and value serialisers and deserialisers. With the error handling deserialiser, you capture the failed message and route it to a dead letter topic for later recovery. You then have access to the failed message and can take the right action to fix and reprocess it.
Overview of the dead letter queue pattern
Validation errors
Validation errors in consumers are another cause of non-transient errors. After deserialisation, a message may fail to meet business criteria or validation rules. In an order processing system, for example, you might see negative amounts or missing customer IDs. These messages should not be processed further and should be sent to a dead letter topic. By throwing a runtime exception and putting the message on the dead letter topic, you make sure the invalid message is captured and can be fixed and reprocessed later.
Add context before you reprocess
To handle non-transient errors well, add contextual information to failed messages: stack traces, the cause of failure, client information and version details. Wrapping the failed message with this information gives useful context for debugging and fixing the issue. It is also good practice to notify the producer team about the failure, so they can take responsibility for fixing and resending the message.
Key takeaway: retry only what can recover on its own; send everything else to a dead letter topic with enough context to fix it and replay it.
How do you retry transient errors in Kafka?
Transient errors are temporary and can be recovered automatically. They occur when an external dependency, such as a database or a service, fails. To handle them, the consumer should retry several times with fixed or incremental intervals in between. This gives the dependency time to come back online so the message can be processed successfully. If the retries fail, the message can be moved to a dead letter topic for further analysis and recovery.
Kafka consumer retry strategy
There are different retry strategies for transient errors. A common approach is to configure the number of retry attempts and the delay between each attempt. In Spring Boot, you can use annotations to configure these settings, such as the number of retry attempts and the delay in milliseconds.
Implementing retries in production can be challenging, especially with high message throughput. There are several variations of retry pattern for handling transient errors effectively:
Blocking retry: the simplest option blocks the consumer thread and retries several times. It is not ideal for high-throughput workloads, because it blocks the main consumer thread and wastes computing resources.
Non-blocking retry topic: an asynchronous version introduces a retry topic. When a transient error is identified, the message is put on the retry topic, isolating the main thread from the retry process. Other scheduling mechanisms then retry the message from the retry topic. If every retry attempt fails, the message moves to the dead letter topic.
Which backoff strategy should you use when reprocessing?
Another approach is exponential backoff, where the delay increases with each retry attempt. This prevents flooding the downstream component with retries. With exponential backoff, you specify a multiplier that sets how much the delay grows on each attempt.
In Spring Kafka, you can configure backoff settings with annotations: the number of retry attempts, the backoff delay and the retry strategy. The strategy can be fixed or exponential, depending on your requirements.
A more advanced non-blocking retry pattern is the leaky bucket pattern. It uses multiple retry topics, such as retry topics 0, 1, 2 and 3, and spreads retry attempts across them. This stops the downstream component from being overwhelmed with retry requests. For more error handling patterns, see our five Kafka error handling tips, and tune the client settings with the Kafka consumer configuring cheatsheet.
How do you build a reliable reprocessing strategy?
Handling failures and errors in Kafka is crucial for building reliable and robust applications. By understanding the different types of error and implementing the right retry mechanisms, you can make sure your application recovers from failures and keeps processing messages effectively. Spring Boot provides convenient tools and configuration for these scenarios, making it easier to build resilient Kafka applications.
What is the difference between transient and non-transient errors in Kafka?
Transient errors are temporary, such as a database or service that is briefly unavailable, and can succeed if you retry. Non-transient errors, such as malformed payloads or messages that fail validation, will fail every time until someone fixes the message or the code.
What is a dead letter topic in Kafka?
A dead letter topic is a separate Kafka topic that holds messages a consumer could not process. Routing failures there keeps the main consumer moving, and the failed messages can be inspected, fixed and reprocessed later.
Should Kafka retries block the consumer thread?
Blocking retries are simple, but they stop the consumer thread and waste resources, so they suit low-throughput workloads. For high throughput, use a non-blocking retry topic so retries run separately from the main consumer.
What is exponential backoff in Kafka consumer retries?
Exponential backoff increases the delay after each failed attempt by a multiplier. It gives the failing dependency time to recover and stops the consumer flooding it with retries. In Spring Kafka you can set the attempts, delay and fixed or exponential strategy with annotations.
What context should you add to a failed Kafka message?
Add the stack trace, the cause of failure, client information and version details. That context makes debugging easier and helps the producer team fix and resend the message.
