Kafka Monitoring and Observability: The Metrics and Tools That Matter
Sion Smith20 October 2026
Blogs9 mins read
Most Kafka outages are visible in the metrics long before anyone raises a ticket. The trouble is volume. A single broker exposes hundreds of JMX MBeans, and every client adds more. This guide covers the small set that tells you whether the cluster is healthy. It also covers how to collect them, which tools to run and what deserves a page at 3am. MBean names below come from the Apache Kafka 4.3 monitoring documentation, the current release line at the time of writing.
Which Kafka metrics matter most?
Start with replication health, controller state, request handling, latency, disk and consumer lag. If those are fine, users are almost always fine. The thresholds below are starting points to tune against your own baseline, not universal rules.
kafka.network:type=RequestMetrics,name=TotalTimeMs for request=Produce and request=FetchConsumer
The request latency clients actually feel, split into queue, local, remote and send time
p99 above your service target for 10 minutes
kafka.log:type=LogManager,name=OfflineLogDirectoryCount plus disk usage
A full or failed disk takes partitions offline
Offline directories above 0; disk above 75 to 80%
Consumer lag (records-lag-max on the consumer, or committed offsets per group)
Consumers falling behind means stale data downstream
Lag growing steadily for 10 minutes or more
Two notes on that table. First, the ISR shrink and expand rates (IsrShrinksPerSec, IsrExpandsPerSec) should sit at zero outside broker restarts, so a steady non-zero rate is worth a ticket. Second, records-lag-max is published by the consumer, not the broker, so you only see it if you collect client metrics. If you are new to how these numbers interact, our guide to Kafka performance basics explains the producer, broker and consumer trade-offs behind them.
How do you collect Kafka metrics?
Kafka publishes its metrics through JMX. You then need something to read them and ship them to a time-series store. There are three common routes, and most production setups use more than one.
1. The Prometheus JMX exporter. This is the default choice for self-managed clusters. It runs as a Java agent inside each broker and serves metrics over HTTP for Prometheus to scrape. Since version 1.x it can also push the same metrics over OTLP to an OpenTelemetry collector, according to the JMX exporter Java agent documentation. A small rules file keeps the output focused:
lowercaseOutputName: true
rules:
# Gauges such as UnderReplicatedPartitions, OfflinePartitionsCount and ActiveControllerCount
- pattern: 'kafka.(server|controller)<type=(.+), name=(.+)><>Value'
name: kafka_$1_$2_$3
type: GAUGE
# p99 request latency for produce and consumer fetch requests
- pattern: 'kafka.network<type=RequestMetrics, name=TotalTimeMs, request=(Produce|FetchConsumer)><>99thPercentile'
name: kafka_network_requestmetrics_totaltimems_p99
labels:
request: "$1"
type: GAUGE
Metrics then appear at http://<broker>:9404/metrics. Use the same agent on your Java producers, consumers and Kafka Streams apps to capture client-side metrics such as lag and error rates.
2. KIP-714 client metrics.KIP-714 lets clients push their metrics to the brokers, so operators can see client behaviour without touching every application. It shipped in Apache Kafka 3.7. It is not automatic, though. The KIP says a metrics plugin implementing the ClientTelemetry interface must be configured on the brokers. At least one client metrics subscription must also be created through the Admin API or kafka-client-metrics.sh. Only then do clients start pushing. KIP-1000 added a way to list those subscriptions. KIP-1076 extends the idea to applications that embed clients, such as Kafka Streams. Our guide to KIP-714 client metrics walks through a working setup.
3. OpenTelemetry. OpenTelemetry complements Kafka’s native metrics rather than replacing them. The OpenTelemetry Java instrumentation for Kafka adds tracing and messaging metrics to clients. Native Kafka client metrics are off by default there and need otel.instrumentation.kafka-clients-metrics.enabled=true. Broker health still comes from JMX.
Which Kafka monitoring tools should you consider?
No single tool covers everything. Most teams pair a metrics stack with a lag checker and a management UI. Each entry below was checked against the project or vendor’s own documentation.
MSK offers CloudWatch and Prometheus open monitoring; the Confluent Cloud Metrics API returns throughput, consumer lag, partition counts and request rates
You only see what the provider exposes
A few practical points. Datadog’s consumer check reads broker high-water marks and committed offsets, then works out lag as the difference. AWS says MSK open monitoring is free, although cross-Availability Zone data transfer is charged. Cruise Control’s compatibility list now includes Kafka 4.0 and 4.3; our Cruise Control guide covers when it is worth running.
For a self-managed cluster, start with the JMX exporter, Prometheus and Grafana. Add Burrow or exporter-based lag metrics, and you have the essentials at no licence cost.
What should you alert on?
Alert on symptoms that users or downstream systems feel. Everything else belongs on a dashboard or in a ticket queue. A noisy pager trains people to ignore it.
Page someone for these:
Offline partitions or under-min-ISR partitions. Data is unavailable or writes are failing right now.
Controller count not equal to 1. The cluster cannot make metadata changes safely.
Consumer lag that keeps growing. Alert on the trend over 10 to 15 minutes, not a single spike after a deploy.
Produce or fetch p99 latency above your service target. This is what producers and consumers actually experience.
Disk headroom running out. Page early enough to add capacity or cut retention before a broker stops.
Raise a ticket, rather than a page, for these:
Under-replicated partitions that clear within a few minutes, which often just means a rolling restart.
A steady trickle of ISR shrinks on one broker, which usually points at a slow disk or network path.
Request handler idle time drifting down week on week, which is a capacity planning signal.
Avoid per-broker CPU alerts on their own. Kafka brokers often run hot without any user impact. Pair CPU with latency or request queue time so that an alert means something.
Key takeaway: Watch replication health, latency, disk and consumer lag together, and alert on what users feel rather than every broker metric.
How does tracing fit in?
Metrics tell you that something is slow. Tracing tells you where. In an event-driven system, a single business request can pass through several producers, topics and consumers. Without trace context, you see each hop in isolation.
The OpenTelemetry Java instrumentation propagates trace context from producers through Kafka message headers, and producer propagation is on by default. Consumers pick up that context, so one trace can span services. This is especially useful for spotting where time goes between a produce call and the final consumer. Our guide to tracing across event-driven architectures shows how to wire this up with Kafka.
Treat tracing as the third layer. Broker metrics show cluster health, client metrics and lag show pipeline health, and traces explain individual slow or failed flows. If you want help putting the three together, talk to OSO’s Kafka support team. If you would rather not run Kafka yourself, look at our fully managed Kafka service.
Frequently asked questions
What are the most important Kafka metrics?
The most important Kafka metrics are under-replicated, offline and under-min-ISR partitions, plus active controller count. Add request handler idle time, produce and fetch latency, disk usage and consumer lag. Together they show whether data is safe, available and flowing on time.
How do I monitor Kafka with Prometheus?
Run the Prometheus JMX exporter as a Java agent on each broker and point Prometheus at its HTTP endpoint. Add a rules file to keep only the metrics you need, then build Grafana dashboards and Alertmanager rules on top.
What is the best free Kafka monitoring tool?
There is no single best free tool, but Prometheus and Grafana with the JMX exporter is the most common open-source choice. Add Burrow or exporter-based lag metrics for consumer lag, and Kafbat UI if you want a web interface for topics and groups.
What is KIP-714?
KIP-714 is the Kafka Improvement Proposal that lets clients push their own metrics to the brokers. It shipped in Kafka 3.7, but it only works once a broker-side metrics plugin and at least one client metrics subscription are configured.
How do I monitor consumer lag?
Monitor consumer lag by comparing each partition’s latest offset with the group’s committed offset. You can read records-lag-max from the consumer, use Burrow, use a commercial check such as Datadog’s, or use your managed service’s lag metrics. Alert on lag that keeps growing rather than on one-off spikes.
OSO engineers run Apache Kafka in production for banks, retailers and the public sector. Tell us what you are weighing up and we will give you a straight answer.
