Tips for moving from Batch to Real Time data streaming
Sion Smith8 August 2023
Blogs7 mins read
Updated October 2026: We checked this guide against current tooling. We clarified that the JDBC source connector polls tables with queries, while log-based change data capture needs a connector such as Debezium, updated the connector count, and noted that Confluent now recommends Apache Flink rather than ksqlDB for new stream processing workloads.
OSO helps teams migrate batch pipelines to real-time streaming data pipelines on Apache Kafka. This guide covers the challenges you are likely to face when moving from batch to real time, the tools and technologies involved, and the lessons we have learned along the way.
Why move from batch to real-time data streaming?
We now live in a world driven by real-time decisions. Nightly or hourly batch jobs leave decision makers working from stale data, so migrating batch processes to streaming has become necessary to meet business demand for timely, accurate data.
How do you choose the first batch process to migrate?
The first step is to select a candidate batch process that is suitable for conversion to real time. Most organisations have hundreds of batch processes, so OSO uses a scoring framework to help decide which to tackle first.
Next, build a proof of concept to demonstrate the value of the real-time pipeline. In one migration this meant implementing a small part of the existing algorithm in Kafka Streams and showing it produced correct results continuously. A narrow, well-chosen proof of concept builds momentum and stakeholder support for the wider programme.
How do you get legacy data into Kafka?
The next step is migrating data from legacy databases into the Kafka pipeline. Kafka Connect is the standard way to integrate external systems with Kafka, and there are hundreds of ready-made connectors on Confluent Hub and elsewhere. Finding the right connectors and extracting the data you need from hidden legacy databases is usually the biggest challenge.
Approach
How it captures changes
Best for
JDBC source connector
Polls tables with a query, using an incrementing or timestamp column
Databases with a JDBC driver and reliable modified-time or ID columns
Log-based CDC (for example Debezium)
Reads the database transaction log
Capturing every insert, update and delete with low latency
Custom producer
Application code writes events directly to Kafka
Systems without a suitable connector
The JDBC source connector is one of the most used connectors because it works with most self-hosted databases that offer a JDBC driver. It is query-based, so you can write a custom query to shape the data you extract, but it only sees rows when it polls and cannot capture hard deletes. When you need every change as it happens, use a log-based CDC connector such as Debezium instead.
How do you validate a streaming pipeline against the batch output?
Once the pipeline is running and data is flowing, prove that it is accurate. Pull the legacy batch output into Kafka with the same connector approach, then use Apache Flink or Kafka Streams to join it with the output of the new pipeline and run validation checks that compare the two streams. Existing ksqlDB users can do the same with ksqlDB, although Confluent now recommends Flink for new stream processing workloads.
Validation is time-consuming but essential. It surfaces discrepancies so you can fix the algorithm, and the results give stakeholders evidence that the pipeline works as intended before the batch job is retired.
What lessons has OSO learned from batch to real-time migrations?
These are the main lessons from OSO’s migrations:
Think in a streaming mindset. Break the algorithm into processing steps and consider the most efficient way to process data at each step, whether you use Kafka Streams, Apache Flink or ksqlDB. In Kafka Streams, separate processing stages worked well. In ksqlDB, multiple chained queries helped optimise processing.
Use Kafka Connect. Kafka Connect was invaluable for integrating with the legacy architecture and saved a lot of time compared with building custom integrations.
Avoid writing custom connectors where you can. Reading from a database and writing to a Kafka topic looks simple, but offsets, schema changes, retries and corner cases make custom connectors hard to get right. A maintained connector is usually more reliable.
Check architectural constraints. Look beyond the pipeline itself. Constraints in the wider business architecture can make a streaming pipeline impractical or not worth the effort, so confirm that real time is the right answer for each process.
Migrating from batch to real-time data streaming with Apache Kafka is complex but rewarding. By starting with a proof of concept, using Kafka Connect and validating the data, you can migrate your pipelines with confidence and deliver fresher, more accurate data.
Key takeaway: Treat batch to real-time migration as a series of small, proven steps. Score and pick one batch process, prove it with a narrow Kafka Streams or Flink proof of concept, ingest legacy data with Kafka Connect or CDC, and reconcile the streaming output against the batch output before you retire the old job.
Frequently asked questions
How do you migrate a batch pipeline to real-time streaming?
Pick one high-value batch process, build a small proof of concept in Kafka Streams or Apache Flink, ingest the source data into Kafka with Kafka Connect or change data capture, then run both pipelines in parallel and compare their outputs. Retire the batch job once the results match.
Is the Kafka JDBC source connector change data capture?
Not log-based CDC. The JDBC source connector polls tables with a query, using an incrementing ID or timestamp column to find new and changed rows, so it misses hard deletes and intermediate changes between polls. For log-based CDC, use a connector such as Debezium that reads the database transaction log.
Should I use ksqlDB or Apache Flink for stream processing?
For new workloads, Confluent recommends Apache Flink, while ksqlDB remains supported for existing applications. Kafka Streams is also a strong choice when you want stream processing embedded in a Java application without a separate cluster.
How do you validate a streaming pipeline against batch results?
Load the legacy batch output into Kafka, join it with the new pipeline’s output using Flink or Kafka Streams, and run checks that compare keys, counts and values. Track discrepancies, fix the logic and share the results with stakeholders before switching over.
When is real-time streaming not worth it?
When nobody acts on fresher data, when upstream systems only produce data in batches, or when wider architectural constraints mean the extra operational cost outweighs the benefit. Score each batch process on business value and feasibility before migrating it.
