blog by OSO

Master Apache Kafka 101

Josh
Josh 28 February 2023
OSO beginner's guide to Apache Kafka basics covering topics, partitions, producers, consumers and brokers, with white title text on a navy panel and a lime Blogs label.

Updated November 2026: We have restructured this beginner’s guide to the Kafka basics, corrected how consumers, ordering and retention work, and updated it for Apache Kafka 4.x and KRaft.

What is Apache Kafka?

Apache Kafka is an open source, distributed system for event streaming and stream processing, maintained by the Apache Software Foundation. It was originally built at LinkedIn to handle large volumes of data with high throughput and fault tolerance, and became an Apache project in 2011.

Kafka delivers low latency and high fault tolerance, which is why companies such as Netflix, Uber, Airbnb and The New York Times use it for real-time data streams, stream processing and data integration.

In a nutshell, a Kafka cluster is a group of servers, called brokers, that communicate over a high-performance TCP network protocol. Producers publish events to topics on the brokers, the brokers store them in partitions, and consumers subscribe to topics and read the events.

How does Kafka’s publish-subscribe model work?

Kafka uses a publish-subscribe (pub-sub) model. Producers publish messages to topics, and consumers subscribe to the topics they are interested in. Producers don’t need to know about consumers and vice versa, which decouples applications and makes the system easier to scale. This decoupling is the foundation of event-driven architecture.

Kafka consumers pull data: each consumer polls the brokers for new records from the partitions it is assigned, starting from its last committed offset. Because the brokers keep the data on disk and replicate it, a consumer that restarts, or moves to another machine, can carry on from where it left off.

What is the distributed log in Kafka?

Kafka is built on the distributed commit log, often just called the log. A log is an ordered, append-only sequence of records: new records are added to the end, and each record gets the next offset in its partition.

Log propertyWhat it means
Append-onlyRecords are only ever added to the end; existing records are never modified
OrderedEach record has an offset, and offsets increase by one within a partition
TimestampedEvery record has a timestamp; if the producer doesn’t set one, Kafka adds it
RetainedRecords stay for the retention period (7 days by default) or until a size limit, whether or not they have been read
CompactableCompacted topics keep only the latest record for each key instead of deleting by age

Records are immutable, but they are not stored forever: Kafka deletes old log segments when the retention limit passes, or compacts them on compacted topics. Figure 1 shows how offsets work once you have read about producers and consumers.

What do Kafka producers do?

Producers are the client applications that write records to Kafka topics.

  • One or many producers can write to the same topic.
  • Producers only append records; they can’t change or delete existing ones.
  • Producers send records in batches, asynchronously, which is a large part of why Kafka achieves high throughput.

How do Kafka consumers read data?

Consumers are the client applications that read records from Kafka topics.

  • Consumers read “at their own pace” by tracking their offset in each partition, so they know where they last read and where to continue.
  • Many consumers, and many independent applications, can read the same log without affecting each other.
  • For example, if a consumer has processed the record at offset 3, its next read starts at offset 4.
  • Consumers that share a group ID form a consumer group and split a topic’s partitions between them, so each partition is read by one consumer in the group.
Apache Kafka Example of offset with 2 consumers
Fig 1. Example of an offset in a topic

What is a Kafka topic?

A topic is a named stream of related records, a little like a table in a database or a folder of files.

  • Think of a topic as a category, such as products or transactions.
  • Producers write to topics and consumers read from them.
  • A topic has one or more partitions.
  • Offsets are unique within a partition, not across the whole topic.
  • Partitions can live on different brokers, so the load of a single topic is spread across the cluster. This is a big part of what makes Kafka fast.

What are Kafka partitions?

Partitions are how Kafka scales a topic. Each partition is an ordered log stored on a broker, and a topic’s partitions are spread across the brokers in the cluster.

  • Partitions on different machines let you scale horizontally: as you need more capacity, you add brokers and spread partitions across them.
  • A producer can write to many partitions of the same topic, and to many topics.
  • Each partition can be replicated to several brokers so the data survives a broker failure.

In figure 2, one producer writes to different partitions of the same topic. Notice that the offsets differ between partitions: each partition keeps its own sequence, and spreading the records across partitions lets the cluster handle more load.

Apache Kafka: Producer writing to various product topics
Fig 2. Producer writing to different partitions of a topic

Kafka guarantees order within a partition, not across partitions. If records for the same entity must stay in order, give them the same key so they land in the same partition.

How does Kafka decide which partition a record goes to?

The producer’s partitioner chooses a partition for every record:

RecordDefault behaviour
Has a keyThe key is hashed, so records with the same key always go to the same partition and stay in order
Has no keyThe producer fills a batch for one partition, then moves to another (“sticky” partitioning), spreading load evenly over time
Explicit partitionThe producer writes to the partition you name

You can also write a custom partitioner, for example to send every “Bike” product to the same partition. Older Kafka clients used round-robin for records without a key; current clients use sticky partitioning because it produces fuller batches.

Does Kafka lock partitions when producers write?

No. Kafka has no locking between producers. Several producers can write to the same topic and partition at the same time, and the broker appends their batches in the order it receives them.

  • Within a single producer writing to one partition, records keep the order they were sent; the idempotent producer, on by default since Kafka 3.0, keeps that order even when it retries.
  • Records from different producers are interleaved, so Kafka doesn’t guarantee an order between producers.
  • Avoiding locks is part of what gives Kafka its throughput: producers send batches asynchronously to one or many topics without waiting for each other.

What is a Kafka cluster?

A Kafka cluster is one or more servers working together.

  • Each server in the cluster runs Kafka as a broker.
  • Topics and their partitions are spread across the brokers in the cluster.
  • Production clusters run several brokers, typically at least three, so partitions can be replicated and the cluster survives a broker failure.
Several brokers within a cluster
Several brokers within a cluster

What is a Kafka broker?

Each server in a Kafka cluster is called a broker.

  • Brokers store partitions and serve reads and writes from clients.
  • A broker can hold many partitions, from many topics.
  • Brokers can be added to or removed from the cluster, with partitions reassigned between them.
  • Data is kept for the configured retention period, whether or not it has been consumed.
  • Each partition has one leader replica, on one broker, that handles its writes; follower replicas on other brokers copy it. The controller chooses the leaders.

What does the Kafka controller do?

The controller manages the cluster’s metadata: which brokers are alive, which topics and partitions exist, and which broker leads each partition. When a broker fails, the controller elects new partition leaders.

In current Kafka, the controller is a small quorum of controller nodes running KRaft, Kafka’s built-in Raft consensus protocol. One controller is active at a time and the others are hot standbys. Kafka 4.0 removed ZooKeeper completely, so new clusters always use KRaft. Read more in our guide to running Kafka without ZooKeeper.

Where can you learn more Kafka basics?

Apache Kafka is a distributed, fault-tolerant, low-latency event streaming platform used by companies such as Netflix, Uber, Airbnb and The New York Times. Its publish-subscribe model decouples producers from consumers, its distributed commit log keeps data ordered and durable, and partitioning lets it scale horizontally.

Next, read the official Apache Kafka documentation, our Apache Kafka glossary, our guide to what Apache Kafka is and getting started with Apache Kafka on Azure. To see Kafka in a real project, read how Dufry built the case for a Kafka architecture with a proof of concept.

Key takeaway: Kafka basics come down to a few ideas: producers append events to topics, topics are split into ordered, replicated partitions across a cluster of brokers, and consumers pull events at their own pace by tracking offsets. Order is guaranteed per partition, data is kept for a retention period rather than deleted on read, and KRaft controllers manage the cluster in Kafka 4.x.

Frequently asked questions

What is Apache Kafka used for?

Kafka is used to move and process streams of events in real time: activity tracking, log and metrics pipelines, event-driven microservices, change data capture from databases and feeding data into analytics and AI systems.

Does Kafka delete messages after they are consumed?

No. Kafka keeps records for the topic’s retention period, 7 days by default, or until a size limit is reached, whether or not anyone has read them. Compacted topics keep the latest record for each key instead.

Does Kafka guarantee message order?

Only within a partition. Records in one partition are read in the order they were written, but there is no order across partitions. Give related records the same key to keep them in one partition.

What is the difference between a topic and a partition?

A topic is a named stream of related records. A partition is one ordered, append-only log within a topic. Splitting a topic into partitions lets Kafka spread it across brokers and consumers.

Does Kafka still need ZooKeeper?

No. Kafka 4.0 removed ZooKeeper. Clusters now use KRaft, where a quorum of Kafka controller nodes manages metadata with the Raft consensus protocol.

Get started with Apache Kafka

Talk to OSO's Kafka engineers about designing your first topics and partitions, running a proof of concept, and taking Kafka safely into production.

CONTACT US
OSO
Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.