KIP-714 Client Metrics: How Apache Kafka Finally Solved Its Biggest Monitoring Challenge

KIP-714 client metrics transforms Kafka monitoring from a fragmented, team-by-team challenge into a centralised, governance-driven solution that automatically captures telemetry from all clients without any application changes. The OSO engineers have validated this approach across enterprise deployments and can confirm it eliminates the three biggest Kafka operational headaches: performance problems, authentication failures, and resource waste.

Why does Kafka monitoring fall short without client metrics?

We’ve all been there—trying to monitor a Kafka ecosystem that started simple but quickly became a tangled web of technologies. What begins as a few Java producers and consumers evolves into a complex distributed system spanning multiple teams, languages and tools. The OSO engineers have witnessed this transformation countless times across client engagements, and the pattern is always the same: monitoring complexity grows exponentially faster than the actual infrastructure.

Consider a typical enterprise Kafka deployment today. You have Java clients running different versions of the Kafka library, some with optimal configurations, others hastily deployed with defaults. Add Kafka Connect workers managed by the data engineering team, ksqlDB queries written by analysts, and Flink jobs deployed by the streaming team. Throw in some legacy .NET applications, Python microservices and Node.js APIs that all need to integrate with Kafka, and suddenly you’re managing monitoring across a dozen different technology stacks.

Each team operates in their own silo, with their own monitoring preferences and maturity levels. The Java team might have comprehensive JMX metrics feeding into their preferred monitoring stack, whilst the Python team struggles with basic throughput visibility. The operations team finds themselves constantly playing catch-up, trying to standardise monitoring across teams that move at different speeds and have different priorities.

Here’s what most organisations miss: Kafka monitoring problems aren’t actually technology problems—they’re governance problems disguised as technical challenges. After supporting over 30 enterprise Kafka deployments, the OSO engineers can confidently state that Kafka itself is rock solid. The problems invariably lie in application misconfigurations and inconsistent operational practices: producers with compression disabled, consumers with suboptimal batch configurations, or authentication policies that create bottlenecks. The broker metrics might show symptoms, but the disease lies in how individual applications integrate with the platform.

How does KIP-714 centralise Kafka client metrics?

KIP-714 fundamentally changes the monitoring paradigm by introducing centralised telemetry collection at the broker level. Instead of trying to standardise monitoring across dozens of client applications, the solution moves telemetry aggregation to the one place where all client interactions converge: the Kafka broker itself.

The technical implementation involves two new protocol verbs that extend the standard Kafka communication protocol. The push_telemetry verb allows clients to send their internal metrics directly to the broker, whilst get_telemetry_subscriptions enables the broker to inform clients about which metrics should be collected and how frequently.

When a client connects to a KIP-714 enabled broker, it automatically negotiates which telemetry capabilities are available. The broker responds with subscription information that tells the client exactly which metrics to collect and how often to send them. This negotiation happens transparently—existing applications continue to work without modification, but they now participate in the centralised monitoring system.

The OSO engineers configure client metrics subscriptions using the kafka-client-metrics.sh tool:

/opt/kafka/bin/kafka-client-metrics.sh --bootstrap-server broker:9092 \
  --alter \
  --name 'basic_producer_metrics' \
  --metrics org.apache.kafka.producer. \
  --interval 5000

/opt/kafka/bin/kafka-client-metrics.sh --bootstrap-server broker:9092 \
  --describe --name "basic_producer_metrics"

This configuration automatically captures producer metrics like compression ratios, batch sizes and throughput rates from all connected clients without requiring any application changes. There’s no way for applications to opt out accidentally, no configuration drift between environments, and no dependency on individual teams remembering to enable monitoring.

KIP-714 also establishes a common vocabulary for Kafka telemetry via OpenTelemetry. Producer latency means the same thing whether it’s measured by a Java application, a .NET service or a Go microservice. The same telemetry data can feed Prometheus, Jaeger and Elasticsearch, so organisations aren’t locked into a single monitoring vendor.

Which Kafka problems does client telemetry catch?

The real power of KIP-714 becomes apparent when examining how it addresses the three most common Kafka operational challenges that plague enterprise deployments.

Performance problems are the most frequent category the OSO engineers encounter. KIP-714 automatically reports compression ratios for every producer, making it immediately obvious which applications are wasting resources. It also exposes batch size and linger.ms behaviour, so operators can spot applications generating excessive network traffic due to inefficient batching. The OSO engineers have seen cases where fixing these configurations improved application performance by 300% whilst simultaneously reducing broker load.

Authentication and authorisation failures traditionally require correlating broker logs with application logs across multiple systems. KIP-714 provides a centralised view of authentication metrics, including failure rates, retry patterns, and the specific principals, client IP addresses and application versions experiencing problems.

Resource waste often provides the highest return on investment. Applications that don’t enable compression might work perfectly from a functional perspective, but they consume three times more network bandwidth than necessary. Uncompressed messages also consume more storage when multiplied by replication factors. The OSO engineers have helped organisations reduce their Kafka storage costs by 50% simply by enabling compression on applications identified through client metrics.

Client telemetry also supports network topology discovery. By analysing producer and consumer relationships, platform teams can automatically generate topology maps showing how data flows through their systems—useful for impact analysis, capacity planning, unused-topic detection and spotting circular data flows.

What do you need to implement KIP-714?

Implementing KIP-714 requires careful consideration of both technical and organisational factors. The technical requirements are straightforward but non-negotiable: both clients and brokers must be running Kafka 3.7 or later, and the cluster must be operating in KRaft mode rather than the legacy ZooKeeper mode.

The KRaft requirement creates the most significant implementation barrier for many organisations. Whilst KRaft mode has been production-ready since Kafka 2.8, many enterprises still operate ZooKeeper-based clusters due to operational inertia or concerns about migration complexity. The OSO engineers recommend treating KIP-714 implementation as an opportunity to modernise the entire Kafka infrastructure, since the benefits of KRaft mode extend well beyond client metrics.

Applications using Kafka client libraries older than 3.7 won’t participate in the telemetry system, creating monitoring blind spots that might persist for months in organisations with long release cycles. The good news is that client upgrades don’t require application code changes—KIP-714 is automatically enabled when compatible client libraries connect to compatible brokers.

KIP-714 also requires deploying a custom metrics reporter plugin to the broker. Treat the plugin as a platform capability rather than a per-application concern. Because the reporter executes inside the broker’s JVM, poorly written plugins can impact broker performance: forward telemetry asynchronously, avoid synchronous I/O, and include circuit-breaker logic so monitoring outages cannot take down the cluster.

Beyond standard client metrics, KIP-714 supports custom application metrics through registerMetricForSubscription, so business metrics such as order-processing latency can share the same telemetry path as infrastructure metrics.

How does KIP-714 change Kafka governance?

The true value of KIP-714 extends beyond technical monitoring improvements—it enables a fundamental shift in how organisations approach Kafka governance. Instead of reactive troubleshooting when problems occur, platform teams can proactively identify optimisation opportunities and provide guidance to development teams before issues impact business processes.

This transformation changes the relationship between platform teams and application teams from adversarial to collaborative. Rather than finger-pointing when performance problems occur, both teams can examine the same standardised metrics. When platform teams can demonstrate value through concrete metrics and optimisation recommendations, it becomes easier to justify infrastructure investments.

Key takeaway: KIP-714 turns Kafka monitoring from a per-team chore into broker-centralised, standardised client telemetry. Upgrade brokers and clients to Kafka 3.7+ on KRaft, deploy an OpenTelemetry-capable metrics plugin, and you can catch misconfigured compression, batching and authentication across every connected client without changing application code.

Frequently asked questions