What are the key concepts of Apache Kafka

At OSO we are seen as the Kafka experts throughout most of Europe, we are the SAS team you call when you need to understand what is going on with your Apache Kafka clusters. Apache Kafka is widely used in modern systems for real-time data streaming and communication between different services. Apache Kafka powers mission-critical systems as the backbone for lightning-fast data streaming and seamless service-to-service communication.

Its robust architecture chews through massive data volumes with ease, making it the go-to solution for teams who need split-second data processing capabilities.

The aim of this document is to cover the basics and high level concepts which you should understand when adopting an event driven architecture. Below is a simple conceptual diagram of multiple microservices all interacting with each a single Apache Kafka cluster.

Key concepts and workflows

Think of a message as the fundamental building block of Kafka’s world – it’s the atom of data streaming. Like a universal container, a message is simply a string of bytes that Kafka treats as a blank canvas, not caring what artistic masterpiece of data you’ve painted inside.

Sometimes, you can tag your message with a special marker called a key – think of it as a smart postal code that helps guide your messages to specific neighbourhoods called partitions. For maximum efficiency, Kafka bundles messages together like a carefully packed shipping container, grouping similar packages heading to the same destination and partition. Each partition works like a giant scroll – new messages are written at the end, and when you read them, you start at the beginning and work your way through, like reading an ancient manuscript.

While a topic is like a city containing many neighbourhood (partitions), remember that only within each individual neighbourhood can you guarantee messages will arrive in the exact order they were sent. Partitions are how Kafka provides scalability and redundancy.

Kafka Clients

In Kafka’s ecosystem, clients play two starring roles: producers, the storytellers who create, and consumers, the audience who listens. Producers are like authors, crafting fresh messages and publishing them to their chosen topic.

Consumers are like meticulous readers, devouring messages chapter by chapter, keeping their place with special bookmarks called offsets – ensuring they never lose track of what they’ve read in each storyline.

Think of an offset as nature’s perfect page number – a never-decreasing counter that Kafka stamps on each message as it arrives. Just as a reader can slide their bookmark into place before taking a break, consumers can pause and resume their reading adventure without missing a single word of the story.

Consumers band together in consumer groups – imagine a book club where members divide up chapters of a massive novel. Like well-organised readers, the group ensures no two members accidentally read the same chapter simultaneously. This clever system of ownership means if one book club member falls asleep, their fellow readers smoothly pick up their assigned chapters – ensuring no part of the story goes unread.

Kafka Infrastructure

Think of a Kafka broker as a digital librarian – a standalone guardian that carefully catalogues incoming messages, stamps them with precise locations, and archives them in its vast storage halls. This masterful curator also serves as a reference desk, efficiently retrieving requested volumes for knowledge-hungry consumers.

Like any world-class institution, Kafka’s library system operates as a sophisticated network of interconnected branches. Within this network, each partition answers to a single authority – the leader broker, much like a department head overseeing their specialised collection.

For safeguarding precious data, mirror collections are maintained by follower brokers, standing ready like trained successors to step up if their department head steps down. This careful replication strategy ensures no knowledge is lost – if one library branch closes, another seamlessly opens its doors to protect the continuous flow of information.

While new manuscripts must be submitted to the department head, scholars can access their research materials from either the head librarian or their trusted deputies. Like a modern archive, Kafka doesn’t keep its collections forever – it employs smart preservation policies.

Each library branch follows preset retention guidelines, maintaining its collections based on either time limits or storage space constraints – much like a rotating exhibition schedule.

Message Formats

Kafka can accept many different formats of message as long as they can be serialised down to bytes. Think of Apache Avro as the universal translator in your data embassy – speaking a diplomatic language that all systems understand through serialisation. At the heart of many data kingdoms stands a master blueprint library, the Schema Registry, safeguarding the official templates for all data structures. While this architectural marvel shares the neighbourhood with Kafka, it maintains its own sovereign territory.

Picture the Schema Registry as the central blueprint office, archiving every approved design pattern used to construct messages in the Kafka ecosystem. By using well-defined schemas stored in a shared repository, Kafka messages can be interpreted without the need for coordination.