Apache Kafka is a distributed data streaming platform that can publish, subscribe to, store and process streams of records in real time. It is designed to handle data flows from many sources to many consumers and to move very large volumes of data. It does not just move data from point A to point B; it moves data from A to Z and beyond, to every system that needs it, at the same time.
Kafka is the leading technology for event-driven architecture. It helps technical leads and senior developers introduce modern software practices and build scalable, high-performance, data-driven systems.
In this “Kafka for beginners” guide we outline the basics of Apache Kafka and event-driven architecture, explain why companies like LinkedIn and Netflix use Kafka, show how to deploy it step by step and where to find Kafka support.
Updated October 2026: this guide reflects Apache Kafka 4.3, the current release line. ZooKeeper was removed in Kafka 4.0, brokers need Java 17 or later and the quickstart commands below are taken from the official Kafka 4.3 documentation.
What is Apache Kafka?
Apache Kafka is an open-source, distributed event streaming platform that lets you publish, subscribe to, store and process streams of records in real time. It was created at LinkedIn, open-sourced in 2011 and is now a top-level project of the Apache Software Foundation. According to the Kafka project, more than 80% of all Fortune 100 companies use it.
Kafka decouples systems, so many teams can consume structured and unstructured data in a consistent way. Because event-driven systems are more modular, flexible and decoupled than batch-based ones, Kafka is a natural fit for Kappa architectures, where one stream of events serves both real-time and historical processing.
Senior developers also like Kafka because its ecosystem keeps growing, with client libraries for most languages, Kafka Connect connectors for hundreds of systems, Kafka Streams for stream processing and tools that simplify integration and deployment.
What is event-driven architecture?
Event-driven architecture (EDA) is a software design in which components communicate by producing and consuming events, where an event is any significant change of state. For example, a customer clicking “buy now” on your website is an event that triggers a series of downstream, asynchronous activities: payment, stock, shipping and email.
The components that generate events, the producers, do not need to know who or what will consume them. They are loosely coupled with the components that consume events, the consumers. Events flow through a broker, which acts as the central point of communication and stores the events so they can be read on demand.
Benefit
What it means in practice
Flexibility
Components are loosely coupled, so you can change, scale and deploy one without affecting the rest
Scalability
Events are processed asynchronously, so you add more producer or consumer instances to handle more load
Modularity
Each service handles specific event types, so you can add features without modifying existing services
Real-time reactions
Events are processed as they happen, which suits analytics, notifications, monitoring and fraud detection
Integration
Events are a common language, so it is easier to connect internal, partner and legacy systems such as point-of-sale
Kafka uses a publish-subscribe model. Producers write events to Kafka topics, and consumers subscribe to those topics and process the events as soon as they arrive, or later, because Kafka keeps them for as long as you configure.
When a producer sends a message, it is appended to a partition of a topic on one broker, the partition’s leader. Follower replicas on other brokers copy it, so with a replication factor of 3 the data survives the loss of a broker and there is no single point of failure. Producers choose how many replicas must confirm a write with the acks setting, and consumers only see messages that have been fully replicated to the in-sync replicas.
Consumers read messages by subscribing to a topic. It works a bit like a newspaper subscription: once you subscribe, you receive every new message on that topic. Unlike a traditional queue, reading a message does not delete it, so many independent applications can read the same topic, each at its own pace.
Each topic is divided into partitions, which are spread across the brokers. Partitions let consumers work in parallel and let Kafka scale horizontally: to handle more data, you add brokers and partitions.
A key, a value, a timestamp and optional headers describing something that happened
Topic
A named stream of events, such as payments, orders or patient_records
Partition
An ordered, append-only log that holds part of a topic; the unit of parallelism and ordering
Producer
An application that publishes events to topics
Consumer and consumer group
Applications that read events; partitions are shared out between the members of a group
Broker
A Kafka server that stores partitions and serves reads and writes
Controller (KRaft)
Nodes that manage cluster metadata and leader elections using the Raft-based KRaft protocol
Topics
A topic is a particular stream of records, a category name to which records are published. A financial services firm might have a topic called payments, an e-commerce store one called orders and a healthcare provider one called patient_records. Topics keep data according to a retention policy based on time or size, or can be compacted to keep the latest value for each key.
Producers
Producers publish records to Kafka topics. Each record has a key, a value and a timestamp, plus optional headers. By default, records with the same key are hashed to the same partition, which keeps all events for one customer or order in sequence. Records without a key are spread across partitions. The value carries the data you want to send.
Consumers
Consumers subscribe to topics and read their records. They are the counterpart of producers. Consumers in the same consumer group share the partitions of a topic between them, and each group tracks its progress, the offset of the last record it processed in each partition. Kafka stores these offsets on the brokers in an internal topic called __consumer_offsets, so a consumer can restart where it left off.
Brokers
Brokers are the servers that receive, store and serve published records. They handle read and write requests from producers and consumers and write records to disk, replicated across brokers, so data is durable and fault tolerant. In KRaft mode, a small set of controller nodes manages the cluster metadata that ZooKeeper used to hold.
Put these components together and you get very high throughput, horizontal scalability and durability.
According to the Apache Kafka project, more than 80% of Fortune 100 companies use Kafka. Users include leaders in entertainment, financial services, e-commerce and software, such as LinkedIn, Box, Goldman Sachs, Uber, Shopify, Target, Netflix, Tesla and TikTok. Here is a sample of how some companies use it.
Company
How it uses Kafka
LinkedIn
Where Kafka was created. Activity tracking, messaging and metrics on more than 100 clusters handling over 7 trillion messages a day (2019)
Netflix
Syncing studio data about talent, budgets, schedules and payments, plus data analytics
Spotify
Event delivery for real-time recommendations, before moving that pipeline to Google Cloud Pub/Sub
Target
Omnichannel retail: syncing stock and sales across stores and digital channels to reduce out-of-stock listings
Netflix. Netflix uses Kafka to sync changes across its studio applications, covering talent, cash flow, budgets, schedules and payments, and to feed the data analytics behind its recommendations.
Target. Target, which in a 2017 Kafka Summit talk described serving more than 26 million site visitors a month, used Kafka to speed up its digital services, sync data across locations and reduce the number of items listed as out of stock.
Kugu and Dufry. OSO helped Kugu scale its building management platform across Europe and Dufry pilot a real-time view of sales and stock with an Apache Kafka proof of concept.
Use Apache Kafka and event-driven architecture when you want your systems to be more flexible, scalable, responsive or fault tolerant. Common uses include:
Scaling a digital business or platform. Reliable, scalable communication between microservices and real-time website metrics (Drivvn and Kugu).
Managing international supply chains, stock and sales. Kafka’s horizontal scalability suits tracking large numbers of related events around the world (Dufry).
Personalising user recommendations. Real-time ingestion and processing improve the content you show users based on what they view or stream.
Generating new value from data. Streaming data into analytics shows broader trends and helps teams improve services (the Department for Education and 33N).
Monitoring fraud and emergency alerts. The publish-subscribe model captures and reacts to events that could put customers at risk (Detected).
Data integration and change data capture. Kafka Connect moves data between databases, SaaS tools and data warehouses without custom code.
How do you deploy Apache Kafka?
To deploy Apache Kafka you follow a few key steps:
Download and install Kafka. It is free and open source under the Apache License 2.0.
Create topics and choose their partitions and replication factor.
Build a simple producer and consumer.
Start producing and consuming events.
Before you start, map out your strategy and check that your hardware and software meet the requirements. Work out your production requirements, such as expected message throughput, storage capacity and fault tolerance, and talk to Kafka experts. Then consider cluster size, network configuration and data retention, as well as your time, budget and in-house Kafka expertise.
What are the requirements to deploy Apache Kafka?
Requirement
What you need
Disk
Enough fast disk for your data volume × retention period × replication factor
Memory
Enough RAM for the JVM heap and, above all, the operating system page cache that Kafka relies on for fast reads
CPU
More cores help with higher throughput, compression and TLS
Java
Java 17 or later for brokers in Kafka 4.x (Java 17, 21 and 25 are supported); clients and Kafka Streams also run on Java 11
Operating system
Linux for production; macOS and Windows (via WSL or Docker) are fine for development
Network
Low-latency links between brokers and correctly configured listeners for clients
Does Kafka still need ZooKeeper?
No. Kafka used to rely on Apache ZooKeeper to maintain cluster metadata such as the list of brokers and topics, to elect partition leaders and to coordinate state changes between brokers. Kafka replaced it with KRaft, a Raft-based consensus protocol built into Kafka itself. KRaft arrived as early access in Kafka 2.8, became production-ready for new clusters in Kafka 3.3, ZooKeeper mode was deprecated in 3.5 and Kafka 4.0 removed ZooKeeper entirely. Existing ZooKeeper clusters must migrate to KRaft on Kafka 3.x before upgrading to 4.0.
That makes deployment simpler: there is no separate ZooKeeper ensemble to install and operate. In KRaft mode, nodes take the broker role, the controller role or both.
Install and start Kafka
Download the Kafka binary distribution from the Apache Kafka downloads page and extract it. Kafka ships with example configuration files in the config directory. For production you edit server.properties on each node to set its node ID, roles, listeners, log directories and defaults such as replication factor, and you plan backup and disaster recovery from the start.
On a single machine, the official quickstart generates a cluster ID, formats the storage and starts the server:
tar -xzf kafka_2.13-4.3.1.tgz
cd kafka_2.13-4.3.1
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties
bin/kafka-server-start.sh config/server.properties
Alternatively, run the official Docker image:
docker pull apache/kafka:4.3.1
docker run -p 9092:9092 apache/kafka:4.3.1
Test Kafka
Kafka is now running on your machine. In a new terminal, create a topic, produce messages and consume them with the command-line tools. Replace my-topic with the name you want:
Track the health, performance and resource use of your cluster with metrics, logging and alerting. Check message production, consumption and data integrity, and load test so you know the cluster can handle the throughput you need.
Usually a managed service. Instead of setting up and running Kafka clusters yourself, Kafka as a service lets a cloud provider handle the operational details of running Kafka, on a pay-as-you-go basis. The main options are Confluent Cloud, Amazon MSK, Google Cloud Managed Service for Apache Kafka and, on Azure, Confluent Cloud or Azure Event Hubs’ Kafka-compatible endpoint. Each has different features and trade-offs.
Full control over configuration, versions and cost, but you run it
Confluent Cloud. From the company founded by Kafka’s creators, Confluent Cloud is a cloud-native Kafka service with enterprise features that boost developer productivity. It runs on all major clouds and connects securely to your applications through private networking, which makes it a good choice for beginners.
Microsoft Azure. Azure is popular with enterprises that have existing Microsoft investments and offers strong integration with Microsoft’s tools. It has no first-party managed Apache Kafka, so most teams use Confluent Cloud on Azure, the Kafka-compatible endpoint of Azure Event Hubs or self-managed Kafka. See our guide to getting started with Apache Kafka on Azure.
Google Cloud. Google Cloud has strong big data and analytics capabilities and offers Managed Service for Apache Kafka for Kafka workloads that feed data processing and machine learning services. Cloud Pub/Sub is an alternative messaging service with a different architecture and API.
AWS. AWS has many services that work well with Kafka, such as Amazon S3 for storage, EC2 for custom infrastructure and Lambda for serverless event processing. You can self-manage Kafka on EC2 with full control over deployment and configuration, or use Amazon MSK, where AWS runs the infrastructure. A typical MSK setup is:
Create an MSK cluster with the AWS console, CLI, SDKs or infrastructure as code.
Choose the number of brokers, broker type, storage, security settings and networking.
Turn on TLS encryption and client authentication with IAM, SASL/SCRAM or mutual TLS.
Adjust cluster configuration through MSK configurations as needed.
Size EBS storage, or use Express brokers, for the capacity and performance you need.
Monitor health, performance and resource use with Amazon CloudWatch or Prometheus.
Remove manual Kafka deployments with infrastructure as code, infrastructure automation and configuration management. You can automate a Kafka deployment in three steps:
Define your infrastructure as code. Use a tool like Terraform to define the resources your Kafka deployment needs: virtual machines or containers, networking, storage and other dependencies.
Provision the infrastructure. We suggest a platform like Kubernetes to run the resources you defined, with networking and storage attached.
Install and configure Kafka. Use an open-source operator like Strimzi to automate installing and configuring Kafka on Kubernetes. Strimzi pulls the Kafka container images and manages broker configuration, listeners, security, certificates and rolling upgrades.
Docker. Docker is not required to run Kafka, but the official apache/kafka image is a convenient way to run Kafka locally and in CI, packaging Kafka and its dependencies so they behave the same in every environment. Read more: What is Docker? 3 powerful gains
Terraform. Terraform is an infrastructure as code tool for declaratively defining and provisioning infrastructure. You can version control your infrastructure, apply consistent changes and manage dependencies between resources. Read more: Making Terraform smarter
Ansible. If you run Kafka on virtual machines, Ansible simplifies configuration management. Playbooks written in YAML describe the desired state and the tasks to run on each server, such as installing a package, changing a configuration or starting a service.
What are the best practices for running Kafka?
Work with a Kafka specialist and be clear about your requirements. Before you start, decide what you want from Kafka: your goals, the data you need to process and your performance requirements. Specialists will need appropriate access to your systems, and be ready to troubleshoot: Kafka is a distributed system, and things will sometimes go wrong.
Secure your cluster from day one:
Encrypt traffic with TLS between clients, brokers and other components to prevent eavesdropping and protect data in transit.
Authenticate clients with SASL (SCRAM, OAUTHBEARER or Kerberos through GSSAPI) or mutual TLS, so only known applications connect. Authenticated identities also let you apply quotas to throttle noisy clients.
Authorise with ACLs to restrict access to topics, consumer groups and administrative operations, granting each user or application only what it needs.
Use firewalls and network segmentation to allow only the necessary ports and IP ranges. If Kafka must be reached across networks, use private connectivity, a VPN or a secure gateway.
Monitor metrics, logs and audit trails to spot suspicious activity and potential breaches.
Audit your cluster regularly to find and fix vulnerabilities and misconfigurations.
Size your cluster for your expected workload, with headroom for peaks and broker failures.
Choose an efficient serialisation format, such as Avro or Protobuf with a schema registry, or JSON where simplicity matters more than size.
Choose keys and partition counts that spread data evenly across partitions while keeping related events in order.
Monitor CPU, memory, network throughput, disk I/O, under-replicated partitions and consumer lag.
Use a replication factor of 3 and min.insync.replicas=2 for important topics, and plan backup and disaster recovery.
Load test with realistic traffic before go-live.
Keep documentation up to date to help with troubleshooting, maintenance and onboarding.
Follow the official Kafka documentation and keep up with Kafka releases.
How do you get Kafka support and learn Kafka?
What is Kafka consulting?
Kafka consultants help you design Kafka-based architectures, choose the right configuration and optimise clusters for performance and scalability. They work with your organisation to understand its use cases, requirements and challenges.
What types of Kafka support can you get?
Support type
What you get
Technical support
Kafka experts who answer technical questions, advise on best practice and troubleshoot deployment, configuration, performance and integration issues by email, video call or ticket
Performance tuning
Analysis of your configuration, partitioning, replication and other settings to improve throughput, latency and scalability
Upgrades and migration
Planning and running upgrades to newer Kafka versions, including ZooKeeper to KRaft migrations, with guidance on compatibility and impact
Training and education
Upskilling your team so it can maintain, improve and troubleshoot Kafka and event-driven systems over time
How can you learn Apache Kafka?
Read the official documentation. The Apache Kafka documentation covers concepts, architecture, configuration, operations and APIs for beginners and experts alike.
Take online tutorials and courses. Platforms such as Udemy, Coursera and LinkedIn Learning offer Kafka courses from introductory to advanced.
Attend conferences, meetups and webinars. Confluent’s Current conference, which replaced Kafka Summit, and local Kafka meetups feature talks by Kafka experts, use cases and emerging best practice. OSO regularly hosts Kafka meetups. Read more: Kafka Summit London: OSO’s 7-step prep guide
Use community resources. The Apache Kafka website links to the mailing lists and community channels where you can ask questions and learn from others.
You now have an overview of Kafka’s core concepts, architecture, deployment options and best practices. With the steps above you can set up, configure and use Kafka with confidence to meet your organisation’s streaming data needs.
Whether you are an executive, developer or data engineer, Kafka opens up many possibilities for building scalable, resilient, data-driven applications. Explore the ecosystem, try different use cases and use the tools and integrations available to get the most from Kafka.
Key takeaway: Kafka is a durable, partitioned, replicated event log that decouples producers from consumers; start with a managed service or the KRaft quickstart, secure it from day one and plan partitions, replication and monitoring before you scale.
Apache Kafka is software that lets applications send, store and receive streams of events. Producers write events such as orders or clicks to topics, Kafka keeps them in a durable, replicated log across several servers, and any number of consumers can read them in real time or replay them later.
What is Apache Kafka used for?
Kafka is used to move data between systems in real time, connect microservices through events, feed analytics and data warehouses, capture database changes, track website activity and metrics, and detect fraud or trigger alerts as events happen. More than 80% of Fortune 100 companies use it.
Is Apache Kafka a database or a message queue?
Neither exactly. Kafka is a distributed event streaming platform built on a replicated commit log. Unlike a traditional queue, it keeps messages after they are read so many consumers can read them independently, but it is not a general-purpose database for querying data.
Does Kafka still need ZooKeeper?
No. Kafka 4.0 removed ZooKeeper entirely. Cluster metadata is managed by KRaft, a Raft-based consensus protocol built into Kafka that has been production-ready for new clusters since Kafka 3.3. ZooKeeper-based clusters must migrate to KRaft on Kafka 3.x before upgrading to 4.0.
Is Apache Kafka free?
Yes. Apache Kafka is open source under the Apache License 2.0 and free to download and run. You pay for the infrastructure you run it on, or for a managed service such as Confluent Cloud, Amazon MSK or Google Cloud Managed Service for Apache Kafka if you prefer not to operate it yourself.
