With Apache Kafka 4.0’s release, the long-awaited future without ZooKeeper has arrived. This version marks a historic milestone: the first release to completely remove the ZooKeeper dependency. For Kafka engineers and architects, this transition represents both an opportunity and a challenge that can’t be ignored.
After more than a decade of evolution, Kafka’s most significant architectural change is now mandatory for moving forward with newer releases. While the KRaft migration promises substantial benefits in scalability, performance and operational simplicity, the transition requires careful planning and awareness of common pitfalls that have tripped up many engineering teams.
This guide distils our hard-earned lessons from production migrations that OSO has already completed. Whether you’re managing a small development cluster or a mission-critical production environment with thousands of brokers, you’ll find actionable insights to help your migration succeed without the downtime and issues that plague hasty implementations.
Updated October 2026: commands and configuration in this guide have been checked against the Apache Kafka 3.9 ZooKeeper to KRaft migration documentation. Kafka 3.9 is the last release line with ZooKeeper support and the bridge release for migration; current releases are in the 4.x line.
How does KRaft change Kafka’s architecture?
The limitations of ZooKeeper-based Kafka
The traditional Kafka architecture relied on ZooKeeper as an external coordination service for metadata management. While this served Kafka well in its early days, several limitations became increasingly apparent as deployments grew.
Partition ceiling: perhaps the most significant limitation was the practical ceiling of around 200,000 partitions per cluster. This threshold wasn’t arbitrary: it stemmed from ZooKeeper’s performance characteristics when handling large amounts of metadata. As organisations scaled their Kafka deployments, this ceiling became a hard constraint on architectural decisions.
ZooKeeper performance bottlenecks: when a broker failed or restarted in large clusters, recovery time was dominated by ZooKeeper operations. The controller, responsible for reassigning partitions, had to perform thousands of sequential ZooKeeper writes, each requiring consensus across the ZooKeeper ensemble. In very large clusters, this could extend recovery times dramatically.
Technical implementation details: the bottleneck stemmed from several factors in the ZooKeeper-based architecture:
Each partition state change required a synchronous ZooKeeper write operation
ZooKeeper’s consensus protocol requires majority acknowledgement for each write
The controller had to process partition reassignments sequentially
ZooKeeper’s performance degraded as the znode tree grew
Watchers and notifications across thousands of znodes created significant overhead
Administrative complexity: managing two distributed systems (Kafka and ZooKeeper) effectively doubled the operational burden. Each system had its own:
Configuration management requirements
Monitoring needs
Failure modes
Scaling considerations
Backup and disaster recovery procedures
Integration challenges: as a separate Apache project, ZooKeeper evolved on its own timeline, sometimes creating version compatibility issues. Kafka had to work within the constraints of ZooKeeper’s design decisions, limiting the improvements that could be made to Kafka’s metadata management.
How KRaft transforms Kafka’s architecture
KRaft (Kafka Raft) implements the Raft consensus protocol natively within Kafka, completely reimagining how metadata is managed and distributed.
The KRaft architecture introduces dedicated controller nodes that form a consensus quorum. These servers run in a dedicated mode focused solely on metadata management. They implement the Raft protocol for distributed consensus and form a quorum that requires a majority to be operational. From among these controllers, a leader is elected that becomes the active controller. Crucially, these controllers operate independently from the data plane brokers.
Controller nodes use a dedicated role (process.roles=controller) and require unique node IDs (node.id) within the cluster. The controller quorum size is typically 3 or 5 nodes: to tolerate N failures you need 2N + 1 controllers. Controllers listen on their own listener (named in controller.listener.names) and keep their own metadata log directory.
Rather than storing metadata in ZooKeeper’s hierarchical structure, KRaft stores metadata in an internal Kafka log, the __cluster_metadata topic, which is hosted on the controller quorum and replicated with the Raft protocol rather than Kafka’s normal replication mechanism. The active controller processes and applies all metadata changes, and snapshots are created periodically so that a node can rebuild its state quickly instead of replaying the whole log. This log-based approach enables much faster recovery during failures or restarts.
KRaft clearly separates the control plane from the data plane. The control plane consists of controllers handling metadata operations and leader elections. The data plane comprises brokers focusing on data storage, replication and client request handling. This separation allows each plane to scale independently based on its resource requirements.
In KRaft, the active controller makes all metadata decisions, with changes committed to the metadata log using Raft. The other controllers replicate this log, while brokers fetch metadata updates from the controllers. This pull-based model reduces controller load compared to ZooKeeper’s push notifications, particularly in large clusters.
Measurable benefits of KRaft
The KRaft architecture delivers several concrete advantages that make the migration worthwhile despite its complexity.
KRaft dramatically improves scalability, raising the practical partition limit from around 200,000 to potentially millions, thanks to more efficient metadata storage and distribution. Test clusters have run with millions of partitions, and metadata operations scale far better with cluster size than they did with ZooKeeper.
Performance gains are equally impressive, with around 10x faster recovery reported for clusters with 2 million partitions. Controlled broker shutdown has been reduced from minutes to seconds, controller failover is much faster, and metadata fetches no longer bottleneck during high-load scenarios.
Area
ZooKeeper mode
KRaft mode
Metadata store
External ZooKeeper ensemble
__cluster_metadata log on the controller quorum
Systems to operate
Kafka and ZooKeeper
Kafka only
Practical partition limit
Around 200,000 per cluster
Millions
Controller failover
Reloads state from ZooKeeper
Hot standby controllers already hold the metadata
Metadata propagation
Controller pushes updates to brokers
Brokers fetch updates from the controllers
Operational simplicity is another major benefit. Rather than managing both Kafka and ZooKeeper, engineers can focus on a single system with unified monitoring, alerting and configuration management. ZooKeeper-specific failure modes are eliminated, and security benefits from a coherent model without cross-system authentication concerns.
Looking to the future, KRaft gives Kafka direct control over the evolution of its metadata protocol and the ability to optimise specifically for Kafka’s needs. This creates a faster development cycle for metadata-related features and a foundation for architectural improvements that weren’t possible with ZooKeeper.
What are the five phases of a ZooKeeper to KRaft migration?
Phase
What changes
Can you revert?
1. Preparation
Upgrade to Kafka 3.9.x, health checks, client inventory
Nothing to revert
2. Controller quorum
Deploy and format dedicated KRaft controllers with migration enabled
Yes: deprovision the controllers
3. Brokers in migration mode
Brokers restart pointing at the controllers; metadata is copied from ZooKeeper
Yes, with the documented ZooKeeper clean-up steps
4. Brokers in KRaft mode
Brokers restart as KRaft brokers; controllers still dual-write to ZooKeeper
Yes, with two rolling restarts and the clean-up steps
5. Finalisation
Controllers leave migration mode; ZooKeeper is decommissioned
No
Preparation and assessment (phase 1)
Successful KRaft migration begins with thorough preparation. This phase is critical yet often rushed, leading to preventable issues later.
The first step involves studying the official Kafka KRaft documentation thoroughly, focusing on the version-specific migration guide, known limitations, the configuration changes required and monitoring recommendations. This groundwork helps teams understand what to expect and prepare accordingly.
Before proceeding, become aware of the key limitations. For migration, controllers must run in dedicated mode (process.roles=controller), not combined mode. You cannot change the metadata version (inter.broker.protocol.version) while the migration is in progress, and features that are not fully implemented in KRaft (such as JBOD with multiple log directories, which was only early access in the 3.x line) can block a migration. Older 3.x releases had further restrictions, so check the Limitations and Missing Features sections for your exact version. If you use JMX over TLS, controllers need the same configuration. ACLs require special attention during migration, and some older clients and tools may have ZooKeeper dependencies that need addressing.
A healthy cluster is essential for a smooth migration. Run health checks to identify and resolve under-replicated partitions, offline partitions, leader imbalances, disk space issues and network latency problems. The following command lists under-replicated partitions:
Ensure the ZooKeeper ensemble is healthy as well (four-letter-word commands must be allowed by 4lw.commands.whitelist):
$ echo ruok | nc zk 2181
Upgrade to the latest Apache Kafka 3.9.x release (the current 3.9 documentation requires at least 3.9.1), or the equivalent commercial version, before migration, and set inter.broker.protocol.version=3.9 on every broker. Plan a rolling upgrade to minimise downtime and verify compatibility with your client applications. Test the upgrade in lower environments first and follow the standard Kafka upgrade process. Confirm all brokers respond on the target version with this command:
Identify all client applications connecting to Kafka and check whether any use ZooKeeper directly. Update clients to use broker APIs instead of ZooKeeper and verify that all clients use bootstrap.servers rather than ZooKeeper connection strings. Test clients against a KRaft cluster in lower environments to ensure compatibility.
Finally, allocate hardware for the controller nodes, typically 3 or 5. Give controllers dedicated, reliable hosts with fast SSD storage for the metadata log and a reliable low-latency network; OSO typically starts production controllers at 8 CPU cores and 32 GB RAM for large clusters, although the Kafka documentation suggests around 5 GB of memory and 5 GB of metadata disk is enough for a typical cluster. Plan your network topology so that controllers can communicate with all brokers without latency or reliability issues.
Controller quorum and migration mode (phases 2 and 3)
This stage deploys the controller nodes and starts the metadata migration.
For phase 2, find the existing cluster ID. It is in the meta.properties file in any broker data directory, or you can read it from ZooKeeper:
$ bin/zookeeper-shell.sh zk:2181 get /cluster/id
Then create server.properties for each controller, with the core controller settings, the migration settings and your security settings. Here’s an example configuration:
controller.quorum.voters must list all controller nodes in the format id@host:port, where id matches each controller’s node.id; Kafka 3.9 also supports controller.quorum.bootstrap.servers for a dynamic quorum, which is what the official example uses. Controller node IDs must not clash with any existing broker.id, because brokers and controllers share one ID namespace in KRaft. The ZooKeeper connection string must match what the brokers use.
Next, format the controller storage with the existing cluster ID:
$ bin/kafka-storage.sh format \
--cluster-id <cluster-id> \
--config server.properties
Start the controller nodes, check their logs for errors and confirm that the quorum has elected a leader:
In phase 3, put the brokers into migration mode. Keep inter.broker.protocol.version=3.9, make sure every broker has an explicit broker.id, and add the KRaft settings to every broker’s properties:
Perform a rolling restart of the brokers, restarting each one and verifying it rejoins the cluster before moving to the next. Once the last broker has restarted in migration mode, the migration starts automatically. When it is complete, the active controller logs this INFO message:
Completed migration of metadata from Zookeeper to KRaft
Use JMX metrics on the controllers to follow the migration state:
Migration has not started; a ZooKeeper controller is in charge
2 (PRE_MIGRATION)
The KRaft controller is waiting for all ZooKeeper brokers to register in migration mode
1 (MIGRATION)
Metadata has been copied to KRaft; the controller is dual-writing to ZooKeeper
3 (POST_MIGRATION)
The migration has been finalised
0 (NONE)
The cluster was created in KRaft mode
Moving the brokers to KRaft (phase 4)
This phase is the critical transition where the brokers stop using ZooKeeper.
Edit every broker’s properties: replace broker.id with node.id (keeping the same ID), add process.roles=broker, and remove the ZooKeeper and migration settings and the IBP, which KRaft replaces with the metadata.version feature:
If you set control.plane.listener.name, remove it. If brokers use kafka.security.authorizer.AclAuthorizer, switch authorizer.class.name to org.apache.kafka.metadata.authorizer.StandardAuthorizer at this point.
Perform another rolling restart of the brokers, checking health before moving to the next one. The controller metric kafka.controller:type=KafkaController,name=MigratingZkBrokerCount should fall to 0 once every broker is running in KRaft mode.
Thoroughly verify cluster health by checking for under-replicated partitions, monitoring consumer lag, verifying producer throughput, validating client connectivity and confirming that topic retention and compaction are working as expected.
This stage is a critical point in the migration. The brokers now talk only to the KRaft controllers, but the controllers are still dual-writing metadata to ZooKeeper, so you can still revert. Reverting is not just a config change, though: it needs two rolling restarts of the brokers and clean-up in ZooKeeper (checking the controller epoch and deleting the /controller and /migration znodes), exactly as described in the “Reverting to ZooKeeper mode during the migration” section of the Kafka documentation. Rehearse it before migration day.
You can see the controllers as voters and the brokers as observers with:
After thorough validation in phase 4, this final phase completes the migration by removing the ZooKeeper dependency from the controllers.
Establish an observation period of at least 7 days in the phase 4 state. During this time, monitor for any unusual behaviour, exercise all client workloads and operations, perform administrative operations to validate functionality and check with all application teams for any issues.
Once you are confident in the stability, remove the migration and ZooKeeper settings from every controller:
Perform a rolling restart of the controllers, one at a time, and check that the quorum keeps a leader after each restart (kafka-metadata-quorum.sh describe --status) and that ZkMigrationState reports 3 (POST_MIGRATION). After this point you can no longer revert to ZooKeeper mode.
After confirming everything is working correctly, decommission ZooKeeper (if nothing else uses it) by backing up its data, shutting down the servers and removing it from your infrastructure.
Perform final validation by verifying all admin operations work, testing broker failure and recovery, validating controller failover and checking that all monitoring and alerting are functional.
What are the common KRaft migration pitfalls?
Pre-migration planning mistakes
Many teams attempt migration on earlier 3.x releases, only to encounter bugs that are fixed in later ones. Typical issues include controller election failures during high metadata load, inconsistent state handling during controller failover, ACL migration failures with large permission sets and metadata migration timeouts with large topic counts. Migrate from the latest 3.9.x release.
Check the release notes (RELEASE_NOTES.html in each Kafka download) for migration-related fixes in newer releases:
Teams often miss KRaft limitations that affect their deployment. Security settings such as SASL mechanisms and TLS for JMX must be configured on the controllers as well as the brokers, and attempting migration with combined mode (controller and broker on the same nodes) is not supported.
KRaft supports two deployment modes: dedicated mode, where controller and broker roles are separated, and combined mode, where both roles run on the same node. For migration you must use dedicated mode, with process.roles=controller on controller nodes and process.roles=broker on broker nodes.
Lack of thorough testing leads to production surprises. Create a comprehensive test plan covering basic operations (produce and consume), administrative operations (topic creation and deletion, configuration changes, ACL management, producer and consumer group management) and failure scenarios (broker failure, controller failure, network partitions, disk failures).
Many applications still use older client libraries or tools with ZooKeeper dependencies. These might include legacy consumer APIs that use ZooKeeper directly, admin scripts that pass --zookeeper, custom monitoring tools that query ZooKeeper, or third-party tools with hard-coded ZooKeeper paths.
Inventory all client applications and check for ZooKeeper dependencies:
Controllers need their own complete configuration: metadata log directories, listener configuration and security settings. If you use TLS, controllers need the same security configuration as the brokers. The controller quorum configuration must include all controllers with matching node ID values.
Network problems are common during migration. These include controllers unable to reach ZooKeeper (port 2181), brokers unable to reach controllers (typically port 9093), controllers unable to communicate with each other or with the brokers’ inter-broker listener, and firewall rules blocking required traffic.
Using different versions for brokers and controllers creates compatibility issues. Controllers on 3.9 with brokers on an older 3.x release may hit protocol mismatches, brokers running mixed versions complicate migration, and ZooKeeper version incompatibilities with older Kafka versions can create unexpected issues.
Default timeouts often fail in geographically distributed deployments. Controller election timeouts may be too short for cross-region latency, quorum request timeouts might be insufficient and socket timeouts may need adjustment for network conditions.
For geographically distributed clusters, consider raising these timeouts (defaults: 1 s, 2 s, 2 s and 30 s), and test the values before production:
The most common process error is insufficient validation between phases. This includes moving to phase 4 before the migration has completed, not verifying broker health after each restart, proceeding despite under-replicated partitions and allowing insufficient time between restarts.
Between phases, inspect the KRaft metadata with the metadata shell, pointing it at the metadata log or a snapshot in a controller’s metadata directory, and list the registered brokers:
$ bin/kafka-metadata-shell.sh \
--snapshot /path/to/metadata/__cluster_metadata-0/00000000000000000000.log
>> ls /brokers
The output should show all brokers registered in the KRaft metadata.
Many teams finalise too quickly (phase 5) before thorough validation. This includes not allowing enough observation time (we recommend 7 days or more), missing weekend or month-end workload patterns, insufficient testing of administrative operations and not validating with all application teams.
Create a validation checklist that includes no under-replicated partitions for 72+ hours, all clients functioning normally, all administrative operations tested (topic creation and deletion, partition reassignment, configuration changes, broker rolling restarts), at least one controller failover tested and all alerting and monitoring systems functioning.
Configuration changes without restarts lead to inconsistent states. This includes changed configs not taking effect, brokers operating in mixed modes and configuration changes applied but not activated.
Lack of visibility into migration progress leads to uncertainty and mistakes. This includes not monitoring the migration state, missing controller election monitoring, no visibility into which brokers have moved to KRaft and an inability to detect failures early.
Set up monitoring for the key controller metrics, including the migration state, active controller count, active brokers and brokers still in ZooKeeper mode:
What are the best practices for a successful KRaft migration?
Comprehensive testing approach
Before attempting a production migration, invest time in thorough testing. Document step-by-step procedures for each phase, include expected outputs and verification steps, create rollback procedures for each phase and test the entire playbook in lower environments.
Test broker failures during migration, controller failures at various phases and network partitions. Create recovery runbooks for each scenario so your team is prepared for any issues that might arise.
Test all client applications against migrated clusters, verify performance characteristics after migration, test administrative tools and scripts, and confirm monitoring systems work with KRaft. This comprehensive client testing helps prevent surprises after migration.
Update dashboards for KRaft-specific metrics, create alerts for controller quorum health, test the observability of controller failovers and make sure operational tools work with the new architecture. Proper monitoring is crucial for both the migration process and ongoing operations.
Migration day execution
When executing the migration in production, monitor migration state transitions, watch for controller elections and monitor broker connectivity to the controllers. These metrics provide visibility into the migration process.
Expect under-replicated partitions during restarts, plan for temporary performance impacts, allow sufficient time for recovery between phases, and don’t abort the migration because of expected transient issues. Understanding what’s normal helps prevent unnecessary rollbacks.
Establish clear communication channels with all stakeholders, schedule regular updates, define escalation paths for issues and keep application teams informed of progress. Communication is often the difference between a migration that is perceived as successful and a problematic one.
Have experts monitoring logs continuously, set up war rooms for each phase, create decision trees for common issues and define clear go/no-go criteria between phases. This preparation helps address problems quickly when they arise.
Contingency planning
No matter how well prepared you are, issues may arise. Document rollback procedures for each phase, test them in lower environments, establish rollback triggers and decision points, and prepare rollback scripts and configurations. Being ready to roll back is essential for risk management.
Consider migrating DR clusters first, maintain the ability to fail over to non-migrated environments, create additional safety nets where possible and use cluster linking or replication for additional protection. These strategies provide fallback options if issues arise.
Define specific metrics for proceeding to the next phase, set maximum allowed degradation thresholds, create unambiguous success criteria and document who has authority to make go/no-go decisions. Clear criteria help prevent rushed decisions during the migration.
Schedule migrations during low-traffic periods, arrange extended team coverage, have vendor support on standby if applicable and allow for extended troubleshooting windows. This planning provides the time and resources needed to address any issues that arise.
Conclusion: embracing Kafka’s KRaft future
The transition from ZooKeeper to KRaft represents more than just a technical migration: it’s an evolutionary leap for Apache Kafka that enables the next generation of features and capabilities. By eliminating the ZooKeeper dependency, Kafka has removed a significant architectural limitation and paved the way for improved scalability, performance and operational simplicity.
As this migration is mandatory for all Kafka users who want to move to version 4.0 and beyond, embracing the change proactively is the wisest course of action. Each organisation’s journey will be unique, but the patterns of success (and failure) are remarkably consistent. By following the approach outlined in this guide, you can navigate this transition with confidence and minimal disruption.
Use this migration as an opportunity to modernise your Kafka operations, clean up technical debt and establish better operational practices. The investment in carefully planning and executing this migration will pay dividends in improved reliability, reduced operational overhead and an infrastructure ready for Kafka’s future innovations.
Begin testing the KRaft migration in lower environments now, inventory and update any client applications that use ZooKeeper, develop a detailed migration runbook specific to your environment and plan training for operations teams on KRaft troubleshooting.
The official migration documentation, community forums, KRaft monitoring guides and the JMX metrics reference are invaluable resources for this journey. Use them alongside the lessons shared in this guide to make your transition as smooth as possible.
Key takeaway: migrate on the latest Kafka 3.9.x with dedicated controllers, validate the dual-write phase for at least a week while you can still revert, and only then finalise and decommission ZooKeeper before upgrading to Kafka 4.x.
Frequently asked questions
Do I have to migrate from ZooKeeper to KRaft?
Yes, if you want to stay current. Apache Kafka 4.0 removed ZooKeeper mode entirely, so every 4.x release runs only in KRaft mode. Kafka 3.9 is the last release line that supports ZooKeeper and the release you migrate from.
Which Kafka version do I need for a ZooKeeper to KRaft migration?
Upgrade all brokers to the latest Kafka 3.9.x release first (the current documentation requires at least 3.9.1) and set inter.broker.protocol.version to 3.9. You can’t upgrade a ZooKeeper-mode cluster straight to Kafka 4.x.
Can I roll back a KRaft migration?
Yes, until you finalise it. While the controllers are still in migration mode and dual-writing to ZooKeeper, you can revert by following the documented steps for the last phase you completed. Once the controllers leave migration mode, you cannot go back to ZooKeeper.
Can I use combined mode controllers during the migration?
No. The migration requires dedicated KRaft controllers with process.roles=controller and separate brokers with process.roles=broker. Combined mode, where one node is both broker and controller, is not supported for migration and is not recommended for production.
How long does a ZooKeeper to KRaft migration take?
The restarts themselves can take hours, but plan for weeks. Allow time for preparation and testing in lower environments, then keep the cluster in the dual-write phase for at least a week, covering weekend and month-end workloads, before you finalise.
