Kafka Cruise Control 101

Kafka Cruise Control from LinkedIn is a powerful open source tool that helps manage and optimise Apache Kafka clusters. We have countless hours of building, configuring and managing 100’s of hours of experience using some of the key features of Cruise Control and how it can benefit your Kafka deployment.

Kafka Cruise Control: Resource Allocation with the Provisioner

When adding a new broker to your Kafka cluster, it is important to consider how resources will be allocated. Cruise Control offers a pluggable framework called the Provisioner, which allows you to define how resources should be allocated. This includes determining if your cluster is under-provisioned or over-provisioned.

Under-provisioning means that your cluster lacks certain resources, while over-provisioning means that you are wasting resources. The Provisioner can help you identify these issues and take appropriate actions, such as removing brokers or rebalancing partitions.

While Cruise Control provides an API for resource allocation, it currently does not have an implementation for specific cloud providers like AWS or Azure. However, you can write your own glue code to integrate with your chosen provider and use the Provisioner API effectively.

Kafka Cruise Control: Automated Rebalancing with the Concurrency Adjuster

Setting manual limits for rebalancing Kafka clusters can be challenging, especially when dealing with fluctuating traffic throughout the day. If the limit is set too high, it can overwhelm the cluster during spikes in traffic. On the other hand, setting the limit too low can result in rebalances that never end.

Cruise Control addresses this issue with its Concurrency Adjuster feature. The Concurrency Adjuster uses metrics such as log flush time, consumer and producer local produce and consume times, and the request queue to determine if the current reassignment limit is appropriate. It uses an additive increase multiplicative decrease algorithm to dynamically adjust the limit based on these metrics.

This automated approach to rebalancing results in faster rebalances without causing spikes or deep issues in the cluster. It also reduces the need for manual intervention, as Cruise Control continuously monitors and adjusts the limits based on the observed metrics.

Kafka Cruise Control: Enhanced Security with Authentication and Authorisation

Since the Log4j vulnerability incident, security has become a top concern for many organisations. Cruise Control offers several authentication and authorization features to ensure the security of your Kafka deployment.

Basic authentication is supported, which works well when used with TLS for secure communication. Additionally, Cruise Control supports Kerberos over HTTP (Aspen Eagle) for distributed authentication. This mechanism allows you to obtain a token and pass it around for communication with the server.

Cruise Control also provides a trusted proxy feature, which helps identify the actual user executing API calls. This is particularly useful when integrating Cruise Control with other products that communicate with each other via APIs. The trusted proxy adds an extra HTTP parameter to the query for authorization purposes.

TLS and HTTPS are available for secure connections to ZooKeeper and Kafka. Cruise Control’s web API is powered by a Jetty web server, making it easy to configure secure connections.

In terms of authorization, Cruise Control offers a simple model with three roles: viewer, user, and admin. The viewer role allows users to view the Kafka cluster and Cruise Control state. The user role grants access to all APIs, while the admin role has the ability to execute commands.

How to manage multiple Kafka Clusters?

Kafka Cruise Control is a valuable tool for managing and optimising Kafka clusters. Its features, such as the Provisioner, Concurrency Adjuster, and authentication/authorization capabilities, help streamline cluster management, improve performance, and enhance security.

By automating tasks like resource allocation and rebalancing, Cruise Control reduces manual labour and ensures stable and efficient cluster operations. Its security features provide peace of mind by protecting your Kafka deployment from unauthorised access and potential vulnerabilities.

If you’re looking to optimise your Kafka deployment and improve its security, consider incorporating Kafka Cruise Control into your workflow. It’s a powerful tool that can help you achieve better performance, stability, and security for your Kafka clusters.