Updated November 2026: We have restructured this guide, updated it for Apache Kafka 4.x (the KStream transform() methods used in the original workaround were removed in Kafka 4.0) and clarified how trace context travels in record headers and how batch spans use links.
What makes up a trace in an event-driven architecture?
A trace is the end-to-end record of one request or event as it moves through your system. It is made up of spans, each representing one unit of work, such as producing a record, processing it in a Kafka Streams topology or writing it to a database from a sink connector. Spans share a trace ID and point to their parent span, so a tracing back end can rebuild the whole journey.
Fig 1. A trace is made up of spans across each step of the event’s journey
In an event-driven architecture, the hard part is carrying the trace context across Kafka. The producer writes the context into the record headers, normally as a W3C traceparent header, and each consumer reads it back to continue the trace.
Building block
What it does
Trace
The full journey of one event, identified by a trace ID
Span
One unit of work, with a start time, duration and attributes
Trace context
The trace ID and parent span ID, carried in the traceparent record header
Span link
A reference from one span to spans in other traces, used when one operation handles many events
What is OpenTelemetry?
The OpenTelemetry project provides a standard, vendor-neutral set of APIs, SDKs and tools for generating and collecting traces, metrics and logs. You instrument your code once and send the data to the back end of your choice.
For Java, the OpenTelemetry Java agent instruments Kafka without code changes. Its supported libraries list includes the Kafka producer and consumer clients, Kafka Streams and Kafka Connect, and the spans follow the OpenTelemetry messaging semantic conventions for Kafka. For clients without the agent, the opentelemetry-kafka-clients library wraps producers and consumers directly.
How do you trace stateless Kafka Streams processing?
Stateless processing in Kafka Streams is straightforward to trace. The Java agent creates a span as each record is processed, continuing the trace from the record’s headers, and the producer instrumentation writes the trace context into the headers of the output records.
If you can’t use the agent, or you need spans around specific steps, you have to start and end spans yourself. Consumer interceptors don’t help much here, because they see whole batches when they are polled, not each record as it moves through the topology. A common workaround is to add a processor node at the start of the topology that extracts the trace context from the record headers and starts a span.
In Kafka 4.x, use process() or processValues() with the Processor API for this node. The older transform() and transformValues() methods were deprecated in Kafka 3.3 (KIP-820) and removed in Kafka 4.0. Test the change carefully: process() can change the record key, which marks the stream for repartitioning, so prefer processValues() when you only need the headers. If you use processValues() on Kafka 4.0.0 or 4.1.0, upgrade to 4.0.1, 4.1.1 or later, which fix a processValues() bug (KAFKA-19668).
How do you trace stateful Kafka Streams processing?
Stateful processing is harder, because state stores and caching break the simple one-record-in, one-record-out flow.
Fig 2. Linking traces through a Kafka Streams state store
Wrap the state store. In an aggregation, wrap the state store with tracing logic so that each update links the span for the new value to the span that wrote the previous value. You can then see how a key’s state changed over time and which events changed it.
Use a custom store. Alternatively, write a custom state store, for example around the RocksDB timestamped key-value store, that delegates get() and put() and records trace information on each read and write. This gives an explicit link between the previous and current values. Read more about custom Kafka Streams state stores.
Caching makes this harder still. The Kafka Streams record cache, sized by statestore.cache.max.bytes (10 MB by default; it replaced the deprecated cache.max.bytes.buffering), coalesces repeated updates to the same key before they reach the store and downstream processors. Updates that are absorbed in the cache never pass through the rest of the topology, so their trace context is lost, and the DSL has no hook to capture it. If you need a span for every update, disable caching for that store (for example with Materialized.withCachingDisabled()) and accept the extra writes; see the Kafka Streams memory management guide.
How do you trace Kafka Connect source and sink tasks?
Kafka Connect runs two types of tasks: source tasks, which read from an external system and write to Kafka, and sink tasks, which read from Kafka and write to an external system.
Fig 3. Tracing through Kafka Connect source and sink tasks
Task type
How tracing works
Main challenge
Source task
Extract the trace context from the source system’s headers or payload, then add a traceparent header to each SourceRecord
The producer instrumentation only uses the span in the current context, so the context must be in place before the record is sent
Sink task
Read the traceparent header from each record and create spans as records are written
put() receives a batch of records, so one write can belong to many traces
For source tasks, extend the connector, or add a single message transform, to read trace information from the source system and put it in the record headers. Remember that the standard producer tracing only takes a span from the current context: if there is no span in the context, it won’t link the trace. Extract the context from the headers and make it current, so the producer picks it up.
For sink tasks, the worker passes a batch of records to SinkTask.put(). Single message transforms run on each record before that, but the write to the destination usually happens for the whole batch. Because one span can only have one parent, create one batch span for the write and add span links to the trace of every record in it. The OpenTelemetry Java agent does this for Kafka Connect 2.6 and later: it creates a span around SinkTask.put() with links to each record’s trace. If you need a separate span per record, the connector itself must be trace-aware and create spans as it writes each record.
How do you implement end-to-end tracing in Apache Kafka?
Instrument producers and consumers with OpenTelemetry, using the Java agent or the Kafka client library, so the traceparent header is written and read automatically.
Make sure every service that creates new records from old ones keeps the trace context, including Kafka Streams apps and custom processors.
Decide how to handle state stores and caching in Kafka Streams: wrap stores, link spans and disable caching where every update must be traced.
In Kafka Connect, add the trace context to source records, and use batch spans with span links in sink tasks.
Send traces to a back end that supports span links, and add Kafka metrics alongside them; see our guide to Kafka monitoring and observability.
Key takeaway: End-to-end tracing in Kafka depends on carrying the trace context in record headers at every hop. Stateless clients and Kafka Streams apps trace easily with OpenTelemetry; stateful processing, caching and batched sink writes need wrapped stores, disabled caching or span links.
We have implemented many types of tracing for a large number of clients. Talk to one of the OSO Kafka experts if you need help.
Frequently asked questions
How is trace context passed through Kafka?
In record headers. OpenTelemetry instrumentation writes the W3C traceparent header, with the trace ID and parent span ID, when a record is produced, and reads it when the record is consumed so the next span joins the same trace.
Does OpenTelemetry support Kafka Streams and Kafka Connect?
Yes. The OpenTelemetry Java agent instruments the Kafka producer and consumer clients, Kafka Streams and Kafka Connect, producing spans that follow the messaging semantic conventions. Custom state stores and connectors may still need manual spans.
Why does caching break tracing in Kafka Streams?
The record cache coalesces repeated updates to the same key before they are forwarded downstream, so some updates never pass through the rest of the topology and their trace context is lost. Disable caching on a store if you need every update traced.
How do you trace a batch of Kafka messages?
Create one span for the batch operation and add a span link to the trace of each message in the batch. A span can have only one parent, but it can link to many spans, so every message’s trace still connects to the batch.