Machine Learning with Apache Kafka

With all the attention on large language models, it is easy to forget the power of a small, highly customised model trained on your own data. Combining Apache Kafka with machine learning (ML) gives you a streaming analytics platform that learns from the same events your business already produces.

ML has two main stages: model building and model deployment. Model building covers data ingestion, pre-processing and training. Model deployment covers real-time scoring and predictions. Kafka’s data integration and stream processing capabilities make it a strong fit for both stages, and this guide walks through each one using a connected-car anomaly detection example.

How do you build a streaming architecture that uses machine learning?

A streaming ML architecture on Kafka usually follows the same sequence, whatever the use case:

1. Ingest data from sources such as car sensors into Kafka, for example over MQTT with a Kafka Connect source connector.
2. Pre-process the data with Kafka-native stream processing such as Kafka Streams or Apache Flink.
3. Land the pre-processed data in a store such as Google Cloud Storage or Amazon S3, or read it straight from Kafka, and train the model with a framework such as TensorFlow.
4. Deploy the trained model into a streaming application (Kafka Streams or Flink), or call it through a model server.
5. Keep historical events available for reprocessing and retraining, for example with Kafka tiered storage.
6. Continuously score new events and publish predictions back to Kafka for downstream consumers.
7. Add the consumer applications, error handling, monitoring and compliance controls a production pipeline needs.

The key design choice is to separate model training from model deployment. Training is a periodic, resource-heavy job; deployment has to keep up with live traffic. Kafka sits between them, so each side can scale and fail independently.

How do you ingest and pre-process data for Kafka machine learning?

Before you can train a model, you need to ingest and pre-process the data. Kafka Connect handles ingestion from many sources. For IoT devices, an MQTT source connector such as the Stream Reactor MQTT connector reads MQTT topics and writes them to Kafka. The ingest cluster can run at the edge, in a data centre or in the cloud, and Kafka can replicate data between clusters, so production and analytics workloads stay separate.

Once the data reaches the analytics cluster, Kafka Streams or Apache Flink can filter, transform and anonymise it in real time. For more complex feature engineering, teams often pair a Python framework with these Kafka-native tools. Agree your event formats early: our guides on designing Kafka events and event streams and managing schemas in Apache Kafka cover the contracts that keep training data consistent.

Can AI models consume directly from a Kafka topic for training?

Yes, but most teams use Kafka as the source of training data rather than training directly against a live topic. There are two common approaches.

Land the data, then train. Sink the pre-processed events into a data lake such as Amazon S3, Google Cloud Storage or HDFS, then train with TensorFlow, PyTorch or Spark ML. This gives you reproducible training sets and lets data scientists iterate without touching the production cluster. In our connected-car example, an autoencoder is trained this way to spot unusual vibration spikes in car engines.

Stream straight from Kafka. Here the ML framework consumes from Kafka itself, which removes the extra data lake. TensorFlow I/O provides Kafka datasets for this, including KafkaGroupIODataset, which tracks progress with consumer group offsets. Its own documentation notes two caveats: the streaming datasets are experimental, and once offsets are committed the consumer does not re-read old messages, so you get a single pass over the data unless you start a new consumer group. Check the project’s maintenance status against your TensorFlow version before you depend on it. Alternatively, a plain Kafka consumer can feed batches to any framework, or to an online learning library such as River, which updates a model one event at a time.

import tensorflow_io as tfio

train_ds = tfio.experimental.streaming.KafkaGroupIODataset(
    topics=["vehicle-vibration"],
    group_id="autoencoder-training",
    servers="kafka:9092",
    configuration=["auto.offset.reset=earliest"],
)

Whichever route you choose, retention decides how far back you can retrain. Tiered storage became production-ready in Apache Kafka 3.9, so brokers can offload older log segments to remote storage such as S3 and keep long histories available for replay without growing local disks.

How do you deploy a trained model on Kafka streams?

Model deployment is separate from training and comes in two main patterns.

Embed the model in the stream processor. The Kafka Streams or Flink application loads the model and scores each event in-process. There is no network hop, so latency is low and the model scales with the application’s partitions.

Call a model server. The streaming application sends each event to a server such as TensorFlow Serving or NVIDIA Triton over gRPC or HTTP and writes the response back to Kafka. This keeps model versioning in one place, but every prediction adds a remote call, which can become the bottleneck at high scale. Flink’s asynchronous I/O helps by keeping many requests in flight, and since Flink 2.1, ML_PREDICT lets Flink SQL call registered models directly.

What does a Kafka anomaly detection example look like?

Take anomaly detection in a connected-car platform. Car sensors publish readings to Kafka over MQTT. Flink pre-processes the data, and the cleaned history is used to train an autoencoder in TensorFlow. The trained model is then deployed in Flink, where it scores every new sensor reading and flags vibration spikes that suggest an engine fault.

The streaming query itself stays simple, while Kafka handles the scale: partitioned topics let you add consumers as the fleet grows, and predictions land on their own topic for alerting, dashboards or maintenance systems. If you keep state in Kafka Streams, our guide to custom Kafka Streams state stores shows how, and our Kafka performance best practices cover the monitoring a production pipeline needs. You will find more examples on the OSO GitHub, and our Kafka consulting services team can help you design the pipeline for your own use case.

Key takeaway: Use Kafka as the backbone of your ML pipeline rather than the training engine. Ingest and clean events with Kafka Connect and stream processing, train from a reproducible copy of the data, and score live events inside Kafka Streams or Flink, or through a model server when you need shared, frequently updated models.

Frequently asked questions