Updated November 2026: We have restructured this beginner’s guide, corrected how tombstones, producer batching, consumer settings and replication work, and updated it for Apache Kafka 4.x.
blog by OSO
Apache Kafka Guide: A Beginner’s Guide
Updated November 2026: We have restructured this beginner’s guide, corrected how tombstones, producer batching, consumer settings and replication work, and updated it for Apache Kafka 4.x.
Kafka is not a traditional queue. A queue usually deletes a message once it has been consumed; Kafka stores events in a log and keeps them for the topic’s retention period, 7 days by default, whether or not anyone has read them. Many independent applications can read the same events, each at its own position.
Events in the log are immutable: once written, they can’t be changed. Kafka removes old data in one of two ways, set per topic by its cleanup policy:
| Cleanup policy | What Kafka keeps | Typical use |
|---|---|---|
| delete (default) | Every event until it is older than the retention time or the log exceeds its size limit | Event streams, logs, metrics |
| compact | The latest event for each key; older events with the same key are removed | Current state, such as a customer’s latest address |
On a compacted topic you delete a key by writing a tombstone: a record with that key and a null value. Compaction then removes the older values for the key, and the tombstone itself is removed after a configurable period (delete.retention.ms, 24 hours by default), giving consumers time to see the deletion.
Every topic is split into partitions, which are spread across the brokers in the cluster. Partitions let Kafka scale horizontally and let many consumers read a topic in parallel. For a deeper introduction, read our Kafka basics guide.
Producers write events to topics. The producer decides which partition each event goes to: events with the same key are hashed to the same partition, so they stay in order, and events without a key are spread across partitions. Producers can write to many topics, and they can compress batches of events before sending them.
Consumers read events from topics. They can read from many topics and are usually organised into consumer groups. Within a group, each partition is assigned to one consumer at a time, so the group shares the work of reading a topic. A different consumer group reads the same events independently.
Producers send events in batches, and two settings control the trade-off between latency and throughput:
| Setting | What it controls | Default |
|---|---|---|
| batch.size | Maximum size of a batch for one partition, in bytes | 16384 (16 KB) |
| linger.ms | How long the producer waits for more events before sending a batch that isn’t full | 5 ms since Kafka 4.0 (0 before) |
A batch is sent when it reaches batch.size or when linger.ms expires, whichever happens first. The two values work together: raising linger.ms has little effect if batches are already full, and raising batch.size has little effect if linger.ms is too short for batches to fill. Tune them together, and measure throughput and latency as you go.
The group ID is the most important consumer setting. Consumers with the same group.id share a topic’s partitions; give each application its own group ID so it receives every event.
| Setting | What it does | Default |
|---|---|---|
| group.id | Names the consumer group the consumer joins | None |
| auto.offset.reset | Where to start when the group has no committed offset (earliest or latest) | latest |
| enable.auto.commit | Commits offsets automatically in the background | true |
| max.poll.records | Maximum records returned by one poll | 500 |
| max.poll.interval.ms | Maximum time between polls before the consumer is removed from the group | 300000 (5 minutes) |
| session.timeout.ms | How long the group waits for heartbeats before treating a consumer as failed (classic protocol) | 45000 (45 seconds) |
| request.timeout.ms | How long the client waits for a response to a request | 30000 (30 seconds) |
Kafka 4.0 made the new consumer group protocol (KIP-848) generally available. Set group.protocol=consumer to use it; rebalances are then managed by the broker and are faster and less disruptive. The default is still the classic protocol. Share groups (KIP-932), production-ready from Kafka 4.2, add queue-style consumption, where several consumers can work on the same partition and acknowledge records individually.
Kafka Connect is Kafka’s configuration-based integration framework. Instead of writing code, you deploy connectors that move data between Kafka and other systems, such as databases, cloud object stores, message queues and document stores like MongoDB.
You configure and start a connector by sending JSON to the Connect REST API, for example with curl. MongoDB’s Kafka connector documentation shows a complete example.
Kafka Streams is a Java library, built on Kafka’s producer and consumer clients, for processing events as they arrive. Your application reads from topics, transforms the events and writes the results back to Kafka, with operations such as filters, joins, aggregations and windowing.
For example, an online shop could join a stream of orders with a stream of payments to find orders that haven’t been paid within 30 minutes, or count orders per product in five-minute windows. Kafka Streams keeps state in local state stores backed by Kafka topics, so it can recover after a failure. Read more about custom Kafka Streams state stores.
A schema registry is a separate component, used alongside producers, consumers, Kafka Connect and Kafka Streams. It stores the schemas (for example Avro, Protobuf or JSON Schema) that describe each topic’s events, and checks that new schema versions stay compatible. That gives teams an agreed contract for their data and stops incompatible events reaching consumers.
| Advantage | Why it matters |
|---|---|
| Scalability | Partitions spread a topic across brokers, so you add brokers and partitions as data grows |
| Fault tolerance | Each partition can be replicated to several brokers; production clusters commonly use a replication factor of 3 (the broker default, default.replication.factor, is 1) |
| Durability and replay | Events are kept for the retention period, so consumers can re-read history or recover from mistakes |
| Real-time processing | Events are available to consumers within milliseconds of being written |
| Decoupling | Producers and consumers don’t need to know about each other, which suits event-driven architecture |
Kafka gives you producers and consumers for writing and reading events, Kafka Connect for integration and Kafka Streams for processing. Good next steps:
Key takeaway: Kafka for beginners comes down to a few ideas: events are stored in replicated, partitioned logs; producers batch and key events; consumer groups share partitions; Kafka Connect integrates other systems and Kafka Streams processes events in real time.
Not in the traditional sense. Kafka stores events in a log for a retention period instead of deleting them when they are read, so many applications can read the same events. Share groups (KIP-932), production-ready from Kafka 4.2, add queue-style consumption on top of that log.
A tombstone is a record with a key and a null value on a compacted topic. It tells Kafka to remove older values for that key during compaction, and is itself removed after delete.retention.ms.
batch.size caps how many bytes the producer puts in a batch for one partition; linger.ms is how long it waits for more events before sending a batch that isn’t full. A batch is sent when either limit is reached.
No, but most production teams use one. A schema registry is a separate service that stores event schemas and enforces compatibility, so producers can’t break consumers by changing the format of the data.
No. Kafka 4.0 removed ZooKeeper. Clusters use KRaft, where Kafka’s own controller nodes manage metadata. Read more in our guide to running Kafka without ZooKeeper.
Talk to OSO's Kafka engineers about designing your first topics, tuning producers and consumers, and taking Kafka safely into production.
CONTACT US