With Apache Kafka 4.0’s release, the long-awaited future without ZooKeeper has arrived. This version marks a historic milestone – the first release to completely remove ZooKeeper dependency. For Kafka engineers and architects, this transition represents both an opportunity and a challenge that can’t be ignored.
After 13 years of evolution, Kafka’s most significant architectural change is now mandatory for moving forward with newer releases. While the KRaft migration promises substantial benefits in scalability, performance, and operational simplicity, the transition requires careful planning and awareness of common pitfalls that have tripped up many engineering teams.
This guide distills our hard-earned lessons from production migrations that OSO already successfully completed. Whether you’re managing a small development cluster or a mission-critical production environment with thousands of brokers, you’ll find actionable insights to ensure your migration succeeds without the downtime and issues that plague hasty implementations.
1. Understanding the KRaft Architecture Revolution
1.1 The Limitations of ZooKeeper-Based Kafka
The traditional Kafka architecture has relied on ZooKeeper as an external coordination service for metadata management. While this served Kafka well in its early days, several limitations became increasingly apparent as deployments grew:
Partition Ceiling: Perhaps the most significant limitation was the 200,000 partition ceiling. This threshold wasn’t arbitrary – it stemmed from ZooKeeper’s performance characteristics when handling large amounts of metadata. As organisations scaled their Kafka deployments, this ceiling became a hard constraint on architectural decisions.
ZooKeeper Performance Bottlenecks: When a broker failed or restarted in large clusters, recovery time was dominated by ZooKeeper operations. The controller, responsible for reassigning partitions, had to perform thousands of sequential ZooKeeper writes, each requiring consensus across the ZooKeeper ensemble. In large clusters with millions of partitions, this could extend recovery times to hours.
Technical Implementation Details: The bottleneck stemmed from several factors in the ZooKeeper-based architecture:
- Each partition state change required a synchronous ZooKeeper write operation
- ZooKeeper’s consensus protocol requires majority acknowledgment for each write
- The controller had to process partition reassignments sequentially
- ZooKeeper’s performance degraded as the znodes directory structure grew
- Watchers and notifications across thousands of znodes created significant overhead
Administrative Complexity: Managing two distributed systems (Kafka and ZooKeeper) effectively doubled the operational burden. Each system had its own:
- Configuration management requirements
- Monitoring needs
- Failure modes
- Scaling considerations
- Backup and disaster recovery procedures
Integration Challenges: As a separate Apache project, ZooKeeper evolved on its own timeline, sometimes creating version compatibility issues. Kafka had to work within the constraints of ZooKeeper’s design decisions, limiting architectural improvements that could be made to Kafka’s metadata management.
1.2 How KRaft Transforms Kafka’s Architecture
KRaft (Kafka Raft) implements the Raft consensus protocol natively within Kafka, completely reimagining how metadata is managed and distributed.
The KRaft architecture introduces dedicated controller nodes that form a consensus quorum. These specialized brokers run in a dedicated mode focused solely on metadata management. They implement the Raft protocol for distributed consensus and form a quorum that requires a majority to be operational. From among these controllers, a leader is elected that becomes the active controller. Crucially, these controllers operate independently from the data plane brokers.
Controller nodes use a special startup mode (process.roles=controller) and require unique node IDs (node.id) within the cluster. The controller quorum size is typically 3 or 5 nodes, with odd numbers required to avoid split-brain scenarios. Controller nodes communicate on a dedicated port (controller.listener.names) and maintain separate data directories from data brokers.
Rather than storing metadata in ZooKeeper’s hierarchical structure, KRaft stores metadata in special Kafka topics. The metadata is stored in a compacted, replicated Kafka topic hosted only on controller nodes. Changes follow the Raft protocol for consistency, with the active controller processing and applying all metadata changes. This log-based approach enables much faster recovery during failures or restarts.
The metadata is stored in a __cluster_metadata topic that uses a custom storage engine optimized for metadata operations. This topic uses the Raft protocol for replication instead of Kafka’s normal replication mechanism, and records are stored in a specially optimized format. Snapshots are created periodically to enable faster recovery, reducing the time needed to rebuild state from the log.
KRaft architecture clearly separates the control plane from the data plane. The control plane consists of controllers handling metadata operations, leader elections, and authorization decisions. The data plane comprises brokers focusing on data storage, replication, and client request handling. This separation allows each plane to scale independently based on its resource requirements, improving overall efficiency.
In the KRaft architecture, the active controller makes all metadata decisions, with changes committed to the metadata topic using Raft. Other controllers replicate this log asynchronously, while data plane brokers pull metadata updates from controllers. This pull-based model reduces controller load compared to ZooKeeper’s push notifications, particularly in large clusters.
1.3 Measurable Benefits of KRaft
The KRaft architecture delivers several concrete advantages that make the migration worthwhile despite its complexity.
KRaft dramatically improves scalability, increasing the partition limit from 200,000 to potentially millions. This expansion comes from more efficient metadata storage and distribution mechanisms. The architecture provides linear scaling with increased broker count rather than the degrading performance seen with ZooKeeper. Test clusters have successfully run with over 5 million partitions, and metadata operations scale with near-constant latency regardless of cluster size.
Performance gains are equally impressive, with 10x faster recovery for large clusters, as measured on 2 million partition clusters. Metadata operation throughput is substantially improved, and latency for controller operations is reduced significantly. Broker shutdown time has been reduced from minutes to seconds, while controller failover time has decreased by orders of magnitude. Partition reassignment speed is improved dramatically, and metadata fetch operations no longer bottleneck during high-load scenarios.
Operational simplicity is another major benefit. Rather than managing both Kafka and ZooKeeper, engineers can focus on a single system with unified monitoring, alerting, and configuration management. ZooKeeper-specific failure modes are eliminated, and security benefits from a coherent model without cross-system authentication concerns.
Looking to the future, KRaft provides Kafka with direct control over metadata protocol evolution and the ability to optimize specifically for Kafka’s needs. This creates a faster development cycle for metadata-related features and establishes a foundation for future architectural improvements that weren’t possible with ZooKeeper.
2. The Five-Phase Migration Journey
2.1 Preparation and Assessment (Phase 1)
Successful KRaft migration begins with thorough preparation. This phase is critical yet often rushed, leading to preventable issues later.
The first step involves studying the official Kafka KRaft documentation thoroughly, focusing on version-specific migration guides, known limitations and issues, configuration changes required, and monitoring recommendations. This groundwork helps teams understand what to expect and prepare accordingly.
Before proceeding, become aware of key KRaft limitations. For migration, only dedicated mode is supported, not combined mode. Authentication has restrictions as well, with SASL/SCRAM for controllers not supported during migration. If using JMX over TLS, additional configuration is required. ACLs require special attention during migration, and some older clients may have ZooKeeper dependencies that need addressing.
You can check KRaft readiness with the following command:
$ bin/kafka-features.sh --bootstrap-server broker:9092 \
--describe --feature kraft
The output should indicate feature compatibility.
A healthy cluster is essential for smooth migration. Run health checks to identify and resolve under-replicated partitions, offline partitions, leader imbalances, disk space issues, and network latency problems. The following command helps identify under-replicated partitions:
$ bin/kafka-topics.sh --bootstrap-server broker:9092 \
--describe --under-replicated
Ensure the ZooKeeper ensemble is healthy as well:
$ bin/zookeeper-shell.sh zk:2181 ruok
Upgrade to Apache Kafka 3.9 (or the equivalent commercial version) before migration. Plan a rolling upgrade to minimize downtime and verify compatibility with your client applications. Test the upgrade in lower environments first and follow the standard Kafka upgrade process. Confirm all brokers are on the target version with this command:
$ bin/kafka-broker-api-versions.sh --bootstrap-server broker:9092
Identify all client applications connecting to Kafka and check if any use ZooKeeper APIs directly. Update clients to use broker APIs instead of ZooKeeper and verify all clients use bootstrap.servers rather than ZooKeeper connection strings. Test clients against a KRaft cluster in lower environments to ensure compatibility.
Finally, allocate hardware for controller nodes, typically 3 or 5. Controllers require substantial resources: at least 8 CPU cores, 32GB RAM, fast SSD storage, and a reliable low-latency network. Plan your network topology to ensure controllers can communicate with all brokers without latency or reliability issues.
2.2 Controller Implementation (Phase 2-3)
This phase involves deploying controller nodes and beginning the metadata migration process.
For Phase 2, start by configuring controller nodes. Create server.properties for each controller with critical configurations for core controller settings, migration-specific settings, and security settings if applicable. Here’s an example configuration:
# Core controller settings
process.roles=controller
node.id=<unique-id>
controller.quorum.voters=<id>@<host>:<port>,<id>@<host>:<port>,...
# Migration-specific settings
kraft.cluster.id=<existing-cluster-id>
zookeeper.connect=<zk-connection-string>
zookeeper.metadata.migration.enable=true
# Security settings (if applicable)
listeners=CONTROLLER://<host>:9093
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:PLAINTEXT
The controller.quorum.voters must list all controller nodes in the format id@host:port where id matches the node.id. The ZooKeeper connection string must match what data brokers use, and the KRaft cluster ID must be extracted from ZooKeeper using:
$ bin/kafka-storage.sh info --cluster-id \
--config server.properties
Next, format the controller storage with the cluster ID:
$ bin/kafka-storage.sh format \
--cluster-id <cluster-id> \
--config server.properties
Start the controller nodes and verify their startup by checking the logs for successful initialization:
INFO [Controller id=1] Initialized raft client (kafka.server.KafkaRaftServer)
INFO [Controller id=1] metadata.migration: Enabled ZK Migration
INFO [Controller id=1] metadata.migration: Starting ZK to KRaft metadata migration
Monitor for controller election, which is indicated by:
INFO [Controller id=1] Elected as the new active controller
In Phase 3, update the inter-broker protocol in all broker properties files:
inter.broker.protocol.version=3.9
Configure brokers for dual connectivity by adding KRaft migration settings to all broker properties:
# KRaft migration settings
controller.quorum.voters=<id>@<host>:<port>,<id>@<host>:<port>,...
zookeeper.metadata.migration.enable=true
Perform a rolling restart of brokers, restarting each one by one and verifying it rejoins the cluster before moving to the next. Monitor the active controller logs for migration progress:
INFO [Controller id=1] metadata.migration: Metadata migration from ZK to KRaft completed successfully
Use JMX metrics to verify the migration state:
kafka.controller:type=KafkaController,name=ZkMigrationState
The value should transition from “ZK_MIGRATION_STARTED” to “ZK_MIGRATION_COMPLETED”.
Verify broker connectivity by checking logs for successful transitions:
INFO [Broker id=1] Transitioned from MIGRATION to RUNNING
2.3 The Transition Point (Phase 4)
This phase represents the critical transition where data plane brokers stop using ZooKeeper.
Remove ZooKeeper configuration from brokers by editing all broker properties files. Comment out or remove the ZooKeeper connection string and add process roles and node ID:
# Remove or comment out
# zookeeper.connect=<zk-connection-string>
# Add or ensure these properties exist
process.roles=broker
node.id=<broker-id>
Disable migration mode on brokers by updating all broker properties:
# Set migration to false
zookeeper.metadata.migration.enable=false
Configure KRaft-specific settings by adding these to all broker properties:
# KRaft-specific configs
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
Perform another rolling restart of brokers, monitoring health before proceeding to the next. Verify broker connectivity by checking logs for successful controller connections:
INFO [Broker id=1] Initialized and registered broker with KRaft controller
Thoroughly verify cluster health by checking for under-replicated partitions, monitoring consumer lag, verifying producer throughput, validating client connectivity, and confirming that topic retention and compaction are working as expected.
This stage represents a critical point in the migration. Data plane brokers now connect only to KRaft controllers, though controllers still maintain dual-write mode to ZooKeeper. This is the last point where rollback is possible – if issues emerge, you can revert broker configs and restart.
You can verify brokers are connected to KRaft controllers using:
$ bin/kafka-metadata-quorum.sh --bootstrap-server broker:9092 \
--describe
2.4 Final Cutover (Phase 5)
After thorough validation in Phase 4, this final phase completes the migration by removing ZooKeeper dependency from controllers.
Establish an observation period of at least 7 days in the Phase 4 state. During this time, monitor for any unusual behavior, exercise all client workloads and operations, perform administrative operations to validate functionality, and check with all application teams for any issues.
Once confident in the stability, remove ZooKeeper configuration from controllers by editing all controller properties:
# Remove or comment out
# zookeeper.connect=<zk-connection-string>
# Disable migration
zookeeper.metadata.migration.enable=false
Perform a rolling restart of controllers, restarting each one by one. Verify controller operation by checking logs for clean startup:
INFO [Controller id=1] Starting KRaft controller
INFO [Controller id=1] Broker registration and active controller election complete
After confirming everything is working correctly, decommission ZooKeeper by backing up its data, shutting down the servers, and removing it from your infrastructure.
Perform final validation by verifying all admin operations work, testing broker failure and recovery, validating controller failover, and checking that all monitoring and alerting are functional.
3. Common Migration Pitfalls and How to Avoid Them
3.1 Pre-Migration Planning Mistakes
Many teams attempt migration on minimum supported versions (Kafka 3.7) only to encounter bugs that are fixed in later releases. The specific issues include controller election failures during high metadata load, inconsistent state handling during controller failover, ACL migration failures with large permission sets, and metadata migration timeouts with large topic counts.
You can check for migration-related fixes in newer releases with this command:
$ grep -i "KRaft\|zookeeper migration\|metadata migration" \
RELEASE_NOTES.md
Teams often miss critical KRaft limitations that impact their deployment. While SASL/PLAINTEXT works, SCRAM authentication for controllers requires additional configuration and may not work during migration. If using TLS for JMX, controllers require special configuration. Attempting migration with combined mode (controller+broker on same nodes) typically leads to failures.
KRaft supports two deployment modes: dedicated mode (supported for migration) where controller and broker roles are separated, and combined mode (not supported for migration) where roles are combined on the same node. For migration, you must use dedicated mode with process.roles=controller for controller nodes and process.roles=broker for broker nodes.
Lack of thorough testing leads to production surprises. Create a comprehensive test plan covering basic operations (produce/consume), administrative operations (topic creation/deletion, configuration changes, ACL management, producer/consumer group management), and failure scenarios (broker failure, controller failure, network partitions, disk failures).
Many applications still use older client libraries with ZooKeeper dependencies. These might include legacy Consumer APIs that use ZooKeeper directly, AdminClient implementations that fall back to ZooKeeper, custom monitoring tools that query ZooKeeper, or third-party tools with hard-coded ZooKeeper paths.
Inventory all client applications and check for ZooKeeper dependencies:
# For Java applications, check for imports
$ grep -r "org.apache.zookeeper" --include="*.java" /path/to/app
# For configuration files, check for ZooKeeper connection strings
$ grep -r "zookeeper" --include="*.properties" /path/to/configs
3.2 Technical Configuration Errors
Controllers require both controller-specific and broker-specific configurations. Missing broker configurations on controllers include log directories, listener configurations, and security settings. If using TLS, controllers need the same security configurations as brokers. The controller quorum configuration must include all controllers with matching node ID values.
Network problems are common during migration. These include controllers unable to reach ZooKeeper (port 2181), brokers unable to reach controllers (typically port 9093), controllers unable to communicate with each other, and firewall rules blocking required traffic.
Verify connectivity with:
# Test ZooKeeper connectivity
$ telnet zookeeper-host 2181
# Test controller connectivity
$ telnet controller-host 9093
# Check firewall rules
$ sudo iptables -L | grep -E '2181|9093'
Using different versions for brokers and controllers creates compatibility issues. Controllers on 3.9 with brokers on 3.7 may encounter protocol mismatches. Brokers running mixed versions complicate migration, and ZooKeeper version incompatibilities with older Kafka versions can create unexpected issues.
Verify all components are on consistent versions:
# Check broker versions
$ bin/kafka-broker-api-versions.sh --bootstrap-server broker:9092
# Check ZooKeeper version
$ echo status | nc zookeeper-host 2181
Default timeouts often fail in geographically distributed deployments. Controller election timeouts may be too short for cross-region latency, controller-to-broker request timeouts might be insufficient, and socket timeouts may need adjustment for network conditions.
For geographically distributed clusters, increase timeouts:
# Controller timeouts for geo-distributed clusters
controller.socket.timeout.ms=60000
controller.quorum.election.timeout.ms=20000
controller.quorum.fetch.timeout.ms=30000
controller.quorum.append.timeout.ms=30000
3.3 Process and Timing Mistakes
The most common process error is insufficient validation between phases. This includes moving to Phase 4 before confirming all metadata is migrated, not verifying broker health after each restart, proceeding despite under-replicated partitions, and allowing insufficient time between restarts.
Between phases, verify migration state:
$ bin/kafka-metadata-shell.sh --snapshot /path/to/kraft/snapshot \
--command "brokers"
The output should show all brokers registered in KRaft metadata.
Many teams finalize too quickly (Phase 5) before thorough validation. This includes not allowing enough observation time (recommend 7+ days), missing weekend or month-end workload patterns, insufficient testing of administrative operations, and not validating with all application teams.
Create a validation checklist that includes no under-replicated partitions for 72+ hours, all clients functioning normally, all administrative operations tested (topic creation/deletion, partition reassignment, configuration changes, broker rolling restarts), at least one controller failover tested, and all alert and monitoring systems functioning.
Configuration changes without restarts lead to inconsistent states. This includes changed configs not taking effect, brokers operating in mixed modes, and configuration changes applied but not activated.
Verify configuration has been applied:
$ bin/kafka-configs.sh --bootstrap-server broker:9092 \
--entity-type brokers --entity-name <broker-id> --describe
Lack of visibility into migration progress leads to uncertainty and mistakes. This includes not monitoring ZooKeeper migration state, missing controller election monitoring, no visibility into metadata synchronization, and inability to detect failures early.
Set up monitoring for key metrics including ZooKeeper migration state, active controller count, and metadata count:
kafka.controller:type=KafkaController,name=ZkMigrationState
kafka.controller:type=KafkaController,name=ActiveControllerCount
kafka.server:type=KafkaServer,name=MetadataCount
4. Best Practices for KRaft Migration Success
4.1 Comprehensive Testing Approach
Before attempting production migration, invest time in thorough testing. Document step-by-step procedures for each phase, include expected outputs and verification steps, create rollback procedures for each phase, and test the entire playbook in lower environments.
Test broker failures during migration, controller failures at various phases, and network partitions. Create recovery runbooks for each scenario to ensure your team is prepared for any issues that might arise.
Test all client applications against migrated clusters, verify performance characteristics post-migration, test administrative tools and scripts, and confirm monitoring systems function with KRaft. This comprehensive client testing helps prevent surprises after migration.
Update dashboards for KRaft-specific metrics, create alerts for controller quorum health, test observability of controller failovers, and ensure operational tools work with the new architecture. Proper monitoring is crucial for both the migration process and ongoing operations.
4.2 Migration Day Execution
When executing the migration in production, monitor migration state transitions, track metadata replication progress, watch for controller elections, and monitor broker connectivity to controllers. These metrics provide visibility into the migration process.
Expect under-replicated partitions during restarts, plan for temporary performance impacts, allow sufficient time for recovery between phases, and don’t abort migration due to expected transient issues. Understanding what’s normal helps prevent unnecessary rollbacks.
Establish clear communication channels with all stakeholders, schedule regular update intervals, define escalation paths for issues, and keep application teams informed of progress. Communication is often the difference between a perceived successful migration and a problematic one.
Have experts monitoring logs continuously, set up war rooms for each phase, create decision trees for common issues, and define clear go/no-go criteria between phases. This preparation helps address problems quickly when they arise.
4.3 Contingency Planning
No matter how well prepared, issues may arise. Document rollback procedures for each phase, test rollback procedures in lower environments, establish rollback triggers and decision points, and prepare rollback scripts and configurations. Being ready to roll back is essential for risk management.
Consider migrating DR clusters first, maintain ability to failover to non-migrated environments, create additional safety nets where possible, and use cluster linking for additional protection. These strategies provide fallback options if issues arise.
Define specific metrics for proceeding to next phases, set maximum allowed degradation thresholds, create unambiguous success criteria, and document who has authority to make go/no-go decisions. Clear criteria help prevent rushed decisions during the migration process.
Schedule migrations during low-traffic periods, arrange for extended team coverage during migration, have vendor support on standby if applicable, and allow for extended troubleshooting windows. This planning provides the time and resources needed to address any issues that arise.
Conclusion: Embracing Kafka’s KRaft Future
The transition from ZooKeeper to KRaft represents more than just a technical migration—it’s an evolutionary leap for Apache Kafka that enables the next generation of features and capabilities. By eliminating the ZooKeeper dependency, Kafka has removed a significant architectural limitation and paved the way for improved scalability, performance, and operational simplicity.
As this migration becomes mandatory for all Kafka users wanting to move to version 4.0 and beyond, embracing this change proactively is the wisest course of action. Each organization’s journey will be unique, but the patterns of success (and failure) are remarkably consistent. By following the detailed approach outlined in this guide, you can navigate this transition with confidence and minimal disruption.
Use this migration as an opportunity to modernize your Kafka operations, clean up technical debt, and establish better operational practices. The investment in carefully planning and executing this migration will pay dividends in improved reliability, reduced operational overhead, and positioning your infrastructure for Kafka’s future innovations.
Begin testing KRaft migration in lower environments immediately, inventory and update any client applications using ZooKeeper APIs, develop a detailed migration runbook specific to your environment, and plan training for operations teams on KRaft troubleshooting. These proactive steps will help ensure your organization is prepared for a successful migration.
Official migration documentation, community forums for troubleshooting, KRaft monitoring guides, and JMX metrics references are invaluable resources for this journey. Leverage these resources along with the lessons shared in this guide to make your transition as smooth as possible.
