Kafka Updates in June 2023

Hey everyone—Welcome to June’s edition of the Kafka Report!

We had a great Kafka Meetup London sponsored by Sainsbury’s at their headquarters. 

We went to AWS Summit London to see the transformative journey to AI 

We created an ultimate guide to building Kafka-as-a-service inside your organisation which is downloadable as a whitepaper 

We also prepared a quiz to score yourself on how ready you are to run Kafka in production! 

We recently held a Kafka Meetup London event on the 26th June at Sainsbury’s HQ, and we’re excited to share the highlights from two enlightening sessions we were privileged to host.

🎙️ Talk 1: Kafka Enterprise Strategy to Evolve with Demand

John Mille, Principal Cloud Engineer at Sainsbury’s, led an engaging discussion about Sainsbury’s transformation journey adopting Kafka with microservices in an event-driven architecture. He shared invaluable insights from the early days, lessons learnt, and successful governance patterns. This talk aimed to equip attendees with strategies for growth, self-sufficiency tools for developers, and more, showing us how to embrace innovative solutions and leap towards the future.

🎙️ Talk 2: Enforcing Kafka Best Practices

In the second session, Tom Scott, Principal Engineer at Conduktor, brought us on a detective hunt exploring tools and metrics used to identify misconfigurations in clients. Tom’s years of experience in Kafka and data integration were evident as he delved into strategies for addressing discovered issues and shared insights on preventing future occurrences. His talk aimed to tackle common Kafka pain points and help attendees kickstart their data journeys.

Both talks will be released on our YouTube channel, subscribe for the latest updates. We extend our gratitude to both John and Tom for their enriching talks and look forward to more such enlightening sessions at our future meetups. If you want to join our meetup group, do so by clicking this link! 🎉

The past weeks have been a whirlwind, full of learning and insights, as we immersed ourselves in the AWS Summit London 2023. This was a fantastic event, abuzz with innovative talks and noteworthy revelations about the future of AI and cloud computing. 🎉💭

Swami Sivasubramanian’s keynote highlighted the ascent of generative AI, featuring the introduction of innovative tools and services like Amazon Bedrock which hosts Amazon’s own large language model (LLM) called Amazon Titan and other 3rd party models such as AI21, and Stability AI. These are poised to expand the potential of LLM’s, leading us into a promising era in our field. 🚀

A striking point during the summit was the fact that only 30% of AWS customers are currently deploying AI workloads in the cloud. This statistic emphasises the necessity of a robust, comprehensive generative AI strategy, rather than just relying on the latest tools. This strategy hinges on a solid data platform that can scale, provides robust security and governance, and aligns with a definitive data strategy. 📊🔒

Our team found Amazon Bedrock and AWS CodeWhisper particularly intriguing. We are eager to see how these new tools compare to ChatGPT, a trusted staple in our productivity suite. 🛠️👀

AWS underscored three principles for a successful data strategy:

1️⃣ A suite of services that address both current and future data needs.

2️⃣ A solution that unifies all data sources.

3️⃣ Effective data governance that ensures secure, fast data flows.

Apache Kafka aligns well with these principles in our view. Its robust data streaming capabilities, wide-ranging Connector ecosystem, and built-in governance mechanisms make it a compelling choice for our data strategy. Furthermore, the ease of integration between Kafka and AWS’s IAM on AWS MSK bolsters its appeal. 🤝

Absorbing the wealth of knowledge from the AWS Summit 2023, we’re ready to navigate the future. With Kafka as the linchpin of our data platform and a clear vision for our generative AI strategy, we look forward to harnessing new AI workloads. If you’re contemplating a data strategy or constructing a resilient data platform to support generative AI workloads, we’d be pleased to share our insights. Watch this space for more updates! 🙌

See the full keynote from AWS Summit London 2023, here.

Our fresh-off-the-press whitepaper 📖 explores the game-changing role of self-hosted Apache Kafka deployments and self-service platforms in revolutionising the developer experience. Unpacking the significance of GitOps and Kubernetes operator patterns 📊, we highlight their power in constructing scalable, internal Kafka services.