The time is finally here where we can run Kafka without ZooKeeper. We will discuss the limitations of using ZooKeeper for replication, how version skew is handled in Kafka, troubleshooting techniques for Kafka clusters in KRaft mode, and the process of upgrading from ZooKeeper mode to KRaft mode.
Zookeeper stores data in data registers called Znodes. ZNodes are arranged in a filesystem-like structure, the name of the Znode resembles a file path. The broker config property zk.connect tells the broker which ensemble to connect and on what path. When the broker joins the cluster, it creates a distinct Znode determined by broker.id property. The path and address in zk.connect have to be the same for all brokers in the cluster and the broker.id has to be unique.
In ZooKeeper, when creating a new topic, you cannot have 3x replication for a new topic. This limitation often leads people to run a minimum of four nodes, even if they only need three. However, in KRaft mode, the controller remembers the brokers that are temporarily down during a role and puts them into a fence state instead of removing them from the cluster. This allows new replicas to be placed on those nodes if no other nodes are available. As a result, you can have a three-node cluster instead of a four-node one if three nodes are suitable for your load.
To upgrade from ZooKeeper mode to KRaft mode, the following steps are involved:
During a role in Kafka, we need to coordinate any changes to the API or the way we use the existing API to ensure that all brokers have enrolled before implementing the change. In ZooKeeper, the inter-broker protocol is used to control the RPC protocols and features supported by the brokers. However, there are some issues with the current use of the inter-broker protocol:
To address these issues, Kafka introduces metadata versioning in KRaft mode. Each inter-broker protocol version will have a corresponding metadata version. Unlike the inter-broker protocol, the metadata version is dynamically configured at the controller level. This means that changing the metadata version does not require a roll and can be done by invoking a controller API. The controller will prevent version changes if there are brokers that have not been upgraded to support the new version. Additionally, Kafka supports downgrading in KRaft mode, allowing dynamic downgrades from the command line. However, there are two types of downgrades: safe and unsafe. Safe downgrades preserve metadata, while unsafe downgrades may result in metadata loss.
In KRaft mode, the cluster metadata replaces ZooKeeper as the store of record. This means that instead of looking at the metadata in ZooKeeper, we need to focus on the metadata in the Kafka topic. There are several tools available to troubleshoot Kafka clusters in KRaft mode:
When monitoring the quorum in KRaft mode, it is important to look at metrics such as the ref state and metadata offset to understand how they are evolving over time. Additionally, there is an RPC (Remote Procedure Call) that can be made to the controller quorum to retrieve information about the quorum. This RPC provides details about the leader ID, leader epoch of the cluster metadata topic, offsets of each member, and the latest offset fetched by each member. Monitoring these metrics and making RPCs to the controller quorum can help in troubleshooting and ensuring the stability of the quorum.
Kafka’s transition from ZooKeeper mode to KRaft mode brings significant improvements in terms of scalability, reliability, and ease of management. The metadata versioning and downgrade capabilities ensure compatibility and flexibility during the upgrade process. If you would like support in moving your clusters off Zookeeper and onto KRaft please contact us for more information and support.
Fore more content:
How to take your Kafka projects to the next level with a Confluent preferred partner
Event driven Architecture: A Simple Guide
Watch Our Kafka Summit Talk: Offering Kafka as a Service in Your Organisation
Successfully Reduce AWS Costs: 4 Powerful Ways
Kafka performance best practices for monitoring and alerting
Have a conversation with a Kafka expert to discover how we help your adopt of Apache Kafka in your business.
Contact Us