Welcome to the latest edition of the Kafka Report, where we bring you the most recent updates and news from the Kafka community. In this report, we’ll explore some of the key improvements in the latest Kafka version, optimal Kafka setup, Kafka deployment best practices, Kafka performance best practices, Kafka performance tuning, Kafka consumer best practices, Kafka production best practices and Kafka events If you’re looking to optimise Kafka performance, implement best practices, or tune your Kafka deployment, you’re in the right place. We’ll address common questions like:
- What is the optimal Kafka setup?
- What is the optimal batch size for Kafka
- How much RAM does Kafka need?
- What is the recommended Ulimit for Kafka
So, let’s dive in and discover the latest insights and recommendations for maximising Kafka’s potential!
The latest Kafka updates: About optimal Kafka setup and test integrations
💥Kafka 3.3.2 is here, with full documentation on the Kafka project site! They’ve made a few improvements—a OAuth/OIDC provider error response, a KRaft broker heartbeat issue—and fixed at least 22 bugs. We’ve pulled a few of them below if you want to view them in JIRA, but head to the RELEASE_NOTES for the unabridged list.
- [KAFKA-14282] – RecordCollector throws exception on message processing
- [KAFKA-14382] – StreamThreads can miss rebalance events when processing records during a rebalance
- [KAFKA-14532] – Correctly handle failed fetch when partitions unassigned
🔥Apache ZooKeeper upgraded to 3.7.1 to address security vulnerabilities. The community recently released ZooKeeper 3.8.0, but the 3.7x is still the stable version and will be until the six-month grace period is up. Since six months isn’t too long, keep watching the space. Plus: here’s the Jira issue: Upgrade zookeeper if you’d like to explore the ticket in greater detail.
The tales we’ve come across as Kafka experts.
✔️ Drivvn discovered that its batch jobs no longer provided its clients with real-time insight, so its technical team launched Apache Kafka using Confluent’s Terraform Provider. In 6 weeks, the team switched from 10+ database tables to 1 KsqlDB statement. Curious to learn more? Read the case study.
🚀 LinkedIn struggled with unused metadata for Kafka clusters until its engineering team deleted about 20% of its topics with TopicGC. In this article, Hao Geng from LinkedIn explains how the team identified and blocked unused topics to reduce bottlenecks and clear up CPU use. The result: 40% improved performance on client requests.
What we learned from others on the web! 🧠 Find the optimal Kafka setup: Explore this kafka-docker-playground repository to experiment with hundreds of possible Kafka Docker configurations. The playground lets you work locally, automate your component testing, and build personalised examples. Tips and tricks included here.
🙌 Integrate tests into Kafka producers: Scan this blog by software architect Vasil Kosturski to see how he uses “Testcontainers” in C#. He’ll walk you through his production code, test code, and his entire Testcontainers customisation, complete with screenshots of the accompanying code in GitHub. If you need some late-night reading, his other posts explore Kafka Listeners in Docker and Kafka networking via Wireshark.
What all the Kafka players are up to, be it acquisitions or major project milestones.
✔️ Confluent acquired startup Immerok to build out its Apache Flink offerings. According to Confluent CEO Jay Kreps, Flink is the next step in Confluent’s mission to make stream processing the de facto method for data architecture: it’ll allow Confluent customers to more easily develop streaming applications. It’s not here yet—the first version will be released “later this year”, but we’re close. Read more about the acquisition in VentureBeat and in Jay Kreps’s blog.
WE’RE BACK IN PERSON.
💥The London Kafka Meetup returned in 2023, featuring two Kafka talks as well as food, drinks, and networking. We were a little nervous when we agreed to help out with bringing back Kafka Meetup – would people still be interested in attending an in person event on a cold January evening in Central London? Well it turns out that the answer to that is a resounding YES! Around 70 people attended the first meetup in 3 years.
Neil’s talk on bringing the 4 pillars to data mesh together with a central nervous system to support a true enterprise Kafka solution resonated well with the audience. Dare’s passionate talk gave those in the room a fresh perspective on how to think about how they approach optimising latency levels. We will share the recordings of these great talks in the next report. If you’re near London, be sure to join the meetup group to be notified of future events. We hope to host another in April!
🧨 The OSO team are now preparing for Kafka Summit London 2023, organised by Confluent! Save the date! May 16-17, a Tuesday-Wednesday combo to break up the week.
It’s a wide, wide world out there. Good thing you’ve got a whole community of Kafka people to have your back. If you don’t have the answer, someone definitely does.
🤝 Questions? Sign up for the Confluent Community Slack, where you can message to get quick answers about event streaming and platform updates. To dive more deeply into technical topics and contribute to a permanent forum, also sign up for the Community Forum Discourse.
That’s all for now—until v2, at least!
Thanks for reading, stay tuned for the next release. Plus: Is your company or team working on an exciting Kafka project? Send us your blog posts, tweets, stories, and scoops to news@oso.sh! You could be featured in the next Kafka Report.
Fore more content:
How to take your Kafka projects to the next level with a Confluent preferred partner
Event driven Architecture: A Simple Guide
Watch Our Kafka Summit Talk: Offering Kafka as a Service in Your Organisation
Successfully Reduce AWS Costs: 4 Powerful Ways
