How to migrate Kafka to KRaft mode

Updated October 2026: Kafka 4.0 (March 2025) removed ZooKeeper mode, so this migration is now mandatory before you can upgrade. We have updated this guide against the Apache Kafka 3.9 documentation. We corrected the order of the steps (the KRaft controllers start before the brokers are rolled, and there are two broker rolls, not one), added the bridge release requirement and the migration state values, and noted that controllers must run in dedicated mode during the migration.

KRaft (Kafka Raft metadata mode) replaces Apache ZooKeeper as the store for Kafka’s cluster metadata. The metadata lives in an internal log, __cluster_metadata, replicated by a quorum of Kafka controllers using the Raft consensus protocol. That gives you:

• Better scalability: KRaft handles far more partitions per cluster than ZooKeeper-based Kafka.
• Faster recovery: standby controllers already hold the metadata, so controller failover and broker restarts are quicker.
• A simpler platform: one system to deploy, secure and monitor instead of two.

Migrating one cluster is well documented. Migrating a large fleet of clusters is harder, because doing them one at a time by hand does not scale. This guide focuses on how to plan and run the migration in batches, what to monitor and how to finalise safely. For the full single-cluster runbook with every configuration file, read our guide to ZooKeeper to KRaft migration.

Why do you need to migrate Kafka to KRaft?

ZooKeeper mode has an end date that has already passed. Every Kafka release from 4.0 onwards runs only in KRaft mode, so a ZooKeeper-based cluster cannot be upgraded until it has been migrated.

You can’t jump straight from a ZooKeeper cluster to Kafka 4.x. Upgrade to 3.9, migrate to KRaft, then upgrade the KRaft cluster to 4.x. Our post on running Kafka without ZooKeeper in production covers what changes once you get there.

How do you plan a KRaft migration across many clusters?

The key is building confidence before you touch the clusters that matter most. Run the migration in waves and only move on when the previous wave has been stable for long enough:

Automate the configuration changes and rolling restarts, so every cluster in a wave follows the same steps. Keep the time each cluster spends in the middle of the migration short: run the second broker roll soon after the first so the cluster is not left half-migrated for long, then leave it in dual-write mode only for the observation period you planned.

Before each wave, check that every cluster:

• runs Kafka 3.9.1 or later with inter.broker.protocol.version=3.9
• has no under-replicated or offline partitions
• has its broker IDs within reserved.broker.max.id (they become KRaft node IDs)
• has no clients or tools that still talk to ZooKeeper directly

What happens during a ZooKeeper to KRaft migration?

The migration (KIP-866) moves a running cluster in three rolling steps after the controllers are in place. At every step until the last, you can still revert to ZooKeeper.

The controller needs these settings for the migration, alongside its normal KRaft settings:

process.roles=controller
node.id=3000
zookeeper.metadata.migration.enable=true
zookeeper.connect=zk1:2181,zk2:2181,zk3:2181
inter.broker.listener.name=PLAINTEXT

Each broker gets these for the first roll:

inter.broker.protocol.version=3.9
zookeeper.metadata.migration.enable=true
controller.quorum.bootstrap.servers=controller1:9093,controller2:9093,controller3:9093
controller.listener.names=CONTROLLER

Run dedicated controllers for the migration. KIP-866 does not support combined broker and controller nodes while a cluster is being migrated.

For the second roll, each broker keeps its ID: replace broker.id with node.id, add process.roles=broker and remove the ZooKeeper settings. If the brokers use ACLs, this is also when you switch from AclAuthorizer to StandardAuthorizer.

How do you test the KRaft migration locally?

Run the whole migration on a local or throwaway cluster before the first wave. It is the fastest way to check your configs and automation:

1. Start a ZooKeeper-mode cluster. Start ZooKeeper, then the brokers on Kafka 3.9.
2. Create and list topics. Produce and consume some data so you can check it survives the migration.
3. Start the KRaft controller. Format its storage with the existing cluster ID and start it with migration enabled.
4. Roll the brokers into migration mode. Restart each broker with the migration settings and watch for the migration to complete.
5. Roll the brokers into KRaft mode. Restart each broker with the ZooKeeper settings removed.
6. Finalise. Restart the controller without the migration settings, then stop ZooKeeper.
7. Verify. Create a new topic and read back the data you produced in step 2.

Use JMX or your metrics stack throughout to confirm each state change, exactly as you will in production.

Which metrics should you monitor during a KRaft migration?

The active KRaft controller reports the migration’s progress. Watch these metrics on it:

They are all in the KafkaController group, for example:

kafka.controller:type=KafkaController,name=ZkMigrationState
kafka.controller:type=KafkaController,name=MigratingZkBrokerCount

The migration state metric reports a number:

When the metadata copy finishes, the active controller also logs Completed migration of metadata from Zookeeper to KRaft at INFO level. The state moves to MIGRATION at that point, and the brokers are still running in ZooKeeper mode until the second roll. Across a fleet, put these metrics on one dashboard so you can see every cluster’s state at a glance.

How do you handle KRaft migration issues?

Most problems show up as a migration that does not progress or a dual-write that falls behind:

1. The migration does not start. The state stays at PRE_MIGRATION until every broker has registered with migration enabled. Check MigratingZkBrokerCount and the controller log for brokers that are missing or misconfigured.
2. ZooKeeper falls behind. During dual-write the controller writes every metadata change to ZooKeeper as well. Watch ZkWriteBehindLag and ZooKeeper’s own latency, and avoid bulk metadata changes such as mass topic creation while the cluster is in this state.
3. Controllers cannot reach brokers or ZooKeeper. The controller needs network access to ZooKeeper and to the brokers’ inter-broker listener. Open the firewall rules before you start.
4. Version changes mid-migration. Don’t change the inter-broker protocol or metadata version while a cluster is being migrated. It can break the cluster. Upgrade first, migrate, then upgrade again.
5. Failed log directories. A ZooKeeper broker with multiple log directories shuts down on any directory failure during the migration, and can only move to KRaft once the directory is repaired.

Turn on TRACE logging for the migration components on the controllers while the migration runs. It makes these issues much easier to diagnose, and you can turn it off afterwards.

How do you finalise the migration and decommission ZooKeeper?

Finalising is the one step you can’t undo, so don’t rush it:

1. Confirm every broker runs in KRaft mode. MigratingZkBrokerCount should be 0 and all brokers active.
2. Observe the cluster. Many teams leave it in dual-write mode for a week or two, through normal peaks, before finalising.
3. Roll the controllers. Remove the migration flag and the ZooKeeper connection from each controller and restart them one at a time.
4. Shut down ZooKeeper. Once the migration is finalised and nothing else uses the ensemble, decommission it.
5. Verify. Create a test topic and check that producers, consumers and admin tooling work as normal.

After this, upgrade the cluster to Kafka 4.x and raise the metadata.version feature level with kafka-features.sh.

What should you check after the KRaft migration?

Keep monitoring once the migration is complete:

• Broker status: every broker registered, unfenced and serving traffic, with no under-replicated partitions.
• Controller quorum: one active controller and stable leadership; check it with kafka-metadata-quorum.sh describe --status.
• Resource use: CPU and memory on brokers and controllers, to spot any change introduced by the migration.
• Topic and admin operations: creating topics, changing configs and reassigning partitions all work as expected.

What have we learned from KRaft migrations?

• Run dedicated controllers. They are required during the migration and recommended for production afterwards, because they separate the control plane from the brokers.
• Use three or five controllers. A single controller is fine for local testing, but production needs a quorum that survives a failure: three tolerate one, five tolerate two.
• Automate everything. The same scripted steps for every cluster make a fleet migration predictable and repeatable.
• Keep improving the runbook. Feed what you learn in each wave back into the next one.
• Use the community. The Kafka documentation, KIP-866 and the community Slack cover edge cases you may hit.

Key takeaway: Migrate Kafka to KRaft on the latest 3.9.x release before upgrading to 4.x. Start dedicated controllers with migration enabled, roll the brokers twice (into migration mode, then into KRaft mode), watch ZkMigrationState until it reaches POST_MIGRATION, and only finalise and remove ZooKeeper after a stable observation period. Run the fleet in waves, from development to the largest production clusters.

Frequently asked questions