Apache Kafka on Azure | Set Up Confluent Kafka

Apache Kafka is a distributed streaming platform that can be used for handling large streams of data. It has become popular in recent years for event streaming because it offers high throughput, scalability and reliability. In our white paper “Apache Kafka on Azure” we will cover:

How to set up Apache Kafka on Azure (Confluent)
Best practices for your Kafka architecture
Outline a production ready config

We will provide you with the Terraform Scripts you can use to make the process automated and repeatable. Find out more about Terraform basics or how to make terraform smarter.

Why run Apache Kafka on Azure?

There are many reasons to run Apache Kafka on Azure cloud. Below are some of the key benefits:

Ensure flexibility and control of your data
Azure provides good scalability and performance on demand for Kafka deployments
Confluent has developed an Azure integration pack which makes it easy to connect Kafka with other Azure services such as EventHubs and CosmosDB

Fully Managed Kafka vs Self-Hosted Kafka

apache kafka on azure - data in the cloud

Managed services such as Confluent Cloud are ideal for businesses or startups that have smaller IT teams that might not have the time or resources to maintain an evolving platform. As Kafka adoption grows inside your organisation, so do your requirements for operating, maintaining and keeping clusters in a healthy state. Why not pay a monthly subscription and have experts deploy, scale, log and monitor their infrastructure which hosts your Kafka? If you require data sovereignty, custom configuration or guaranteed high performance then you are going to want to host your own. The deployment guide from OSO is a great place to start your self hosted stream processing journey.

What companies use Kafka?

Many companies use Kafka including Uber, Netflix, Pinterest and LinkedIn. If you’re looking for a complete list of companies that use Kafka, check out this link.

Now that we’ve gone over some of the reasons why you might want to run an event driven platform, let’s walk you through what this comprehensive Kafka deployment guide provides.

About this Guide

This deployment guide provides step-by-step instructions for deploying Confluent Enterprise on Azure. It is aimed at cloud architects, administrators and DevOps professionals who are planning to implement or extend their Kafka workloads on Azure with great uptime and low SLA. Confluent Enterprise is available as a bring-your-own-license offering, this guide simplifies deployment of your streaming infrastructure and applications, making it easy to configure, deploy, monitor and manage Confluent clusters. Confluent builds on Kafka with additional tooling to speed developer productivity, real-time improve data management, and broaden the scope of applications that can benefit from the Kafka API.

Kafka clusters on Azure can support 10s to 1000s of traditional producer/consumer workloads as well as the latest Kafka Connect pipelines with unlimited connectors and Kafka clients. This is presented as an alternative to the Azure Marketplace with recommended best practices for running a highly available production environment with other Azure services.

How OSO set up Confluent Kafka to Azure

The recommended way to set up Confluent Apache Kafka on Azure is by using the Terraform Scripts we have provided here. This will allow you to automate the process of provisioning and managing your production-ready Kafka brokers, Confluent Schema Registry, Kafka Connect, Confluent Control Center and Kafka Rest Proxy.

If you are not familiar with Terraform, it is a tool that allows you to define infrastructure as code. This means that you can provision and manage your infrastructure using code rather than manually doing so through the Azure portal.

The Terraform Scripts we have provided will create the following resources in Azure:

A Resource Group to store all the resources related to your Kafka deployment.
An Azure blob Storage Account to store the Terraform state file. This is used by Terraform to track what resources have been created and keep track of any changes made to them.
A Virtual Network and Subnet to store your Kafka brokers.
A Public IP to load balance traffic between your Kafka brokers.
A Public IP Address for the Control Center.
Separate data disks for log storage
Kafka Brokers deployed in an Azure Scale Set. The number of brokers deployed can be configured.
A Confluent Control Center deployment which is used for managing and monitoring your Kafka deployment.
Monitoring integration into Azure App Insights to monitor production cluster.

The Terraform Scripts will also create the following Azure Active Directory resources:

A Service Principal which is used by Terraform to authenticate with Azure.
A Role Assignment which gives the Service Principal permissions to

Our Confluent Relationship

OSO is a Preferred Consulting partner with Confluent, this provides us access to Kafka internal resources, as well as proof of concept funding. We have the people and knowledge that provide distinct and differentiated value for any event driven architectures our customers want to explore.

If you are interested in learning more about Confluent or Apache Kafka on Azure or any of the other services that we offer, please reach out to us. We would be happy to discuss how we can help you achieve your business goals.