Updated November 2026: We have restyled this August 2023 newsletter issue. The batch to real-time steps are now a checklist, the adoption statistics are attributed to their sources, the event notice is now a record of the September 2023 meetup, the related resources point to our current guides, and a retired eBook link has been removed.
Customer 360 is a single, complete view of each customer, so you can tailor offers to their needs. It combines internal data on an individual, such as account details, postcode, age and contact information, with behaviour such as past customer service requests, social media interactions and how they move through your website.
Why it matters: with Customer 360 you can spot when clients are in the market for a particular product. They might move house and downsize, leaving capital to invest. They might show strong interest in a socially responsible investment package. They might buy the same advisory support on the same date each year, so you can anticipate the request.
How Kafka helps: Apache Kafka is an event streaming platform that underpins event-driven architecture (EDA). Every time an event happens, such as a website click or a transaction, it is published to Kafka in real time. Instead of separate systems drifting out of sync, Kafka collects events from many sources so downstream services can build a consistent, up-to-date view of each customer.
With OSO’s Kafka Support, Cambridge University Press added a data streaming pipeline and submission service to enhance its online learning system.
| Area | Cambridge University Press |
|---|---|
| Challenge | Update syllabi, tests and assessments easily, and collect and store accurate data in real time |
| Solution | A reliable data pipeline and submission service, delivered with OSO Kafka Support instead of hiring a full-time Kafka engineer |
| Expected results | A better learning experience, richer assessment feedback and more strategic decisions (the project was in its early phases at the time) |
In the Press’s words: “The successful integration of content from Contentful into our internal systems in real-time improved the overall user experience and data accuracy, setting a strong foundation for future data-related initiatives.”
Read the full Cambridge University Press customer story.
Kafka is now mainstream: the Apache Kafka project says more than 80% of all Fortune 100 companies use it, and Confluent’s chief executive Jay Kreps has said more than 75% of the Fortune 500 use Kafka.
Instead of processing data in batches, or discrete chunks on a schedule, real-time processing publishes events as soon as they happen. A typical migration follows these steps:
| Step | What to do |
|---|---|
| 1. Choose a pilot | Pick one batch process with clear business value and manageable scope for a proof of concept |
| 2. Connect the source | Use Kafka Connect, for example a change data capture (CDC) source connector, to stream changes from your current database into Kafka |
| 3. Build the pipeline | Process events with Kafka Streams, ksqlDB or Flink, and write results to the target system with a sink connector |
| 4. Validate | Run the real-time pipeline alongside the batch job and compare outputs until they match |
| 5. Switch and scale | Retire the batch job, then repeat the pattern for the next process |
First steps: identify the first batch process to migrate using a simple framework. Will proving success on a small scale help you make the case for Kafka to senior leadership? If you are not sure they will listen, position real-time data against your company’s bigger strategic goals.
Why move at all? Three themes came up in this issue:
Key takeaway: Move from batch to real-time one process at a time: pilot a single job, stream its source into Kafka with Kafka Connect, validate the real-time output against the batch run, then retire the batch job.
This issue promoted the Big Data LDN x Kafka Meetup London special event, which has now taken place:
| Detail | Event information |
|---|---|
| Date and time | Wednesday 20 September 2023, 18:00 to 21:00 |
| Venue | Fast Data Theatre, Big Data LDN, Olympia, London W14 8UX |
| Theme | Exploring Fast Data with Kafka: Moving from Batch to Real-Time |
| Speakers | Quix and Aerospike, with Q&A hosted by OSO |
The three-hour event combined talks on real-time data, networking, pizza and Q&A sessions. Read the Big Data LDN x Kafka Meetup London 2023 recap for the speakers and talks.
We added more guides on migrating batch processes to real time. These are the most relevant:
Kafka Support is dedicated enterprise support for Apache Kafka. Instead of hiring a full-time Kafka engineer, you work with a team of Kafka experts who help implement and troubleshoot your setup.
| Feature | What you get |
|---|---|
| Response time | A service level agreement committing to respond within 2 business days |
| Channel | A Slack channel to talk directly with OSO engineers |
| Scope | Your whole streaming stack, from Linux volumes and broker configuration to application code |
| Cost | A fraction of the cost of hiring a single Kafka engineer full time |
How to prepare: consider the level of support you need and your budget. If you need a plan for senior leadership, OSO can help you outline the outcomes of real-time data, such as faster fraud detection, a 360-degree customer view and more accurate data for reporting and compliance. See Kafka Support or contact us.
Start with one batch process as a pilot. Stream its source data into Kafka with Kafka Connect, process the events with Kafka Streams, ksqlDB or Flink, validate the output against the batch job, then retire the batch job and repeat.
A pipeline that moves and processes each event as soon as it happens, instead of on a schedule. With Kafka, source connectors publish events to topics, stream processors transform them and sink connectors deliver the results.
Customer behaviour changes quickly. Streaming events such as clicks, transactions and service requests into Kafka keeps each customer profile current, so offers and service can respond to what is happening now.
The August 2023 issue covered Customer 360, the Cambridge University Press Kafka Support story, how to move from batch to real-time, the Big Data LDN x Kafka Meetup London event and new batch to real-time resources.
Kafka Support customers have a service level agreement in which OSO commits to respond to requests within 2 business days, with a Slack channel for talking directly to OSO engineers.
Talk to OSO's Kafka engineers about piloting, building and running your first real-time data pipeline on Apache Kafka.
CONTACT US