Presentations from Kafka Meet Up London January 2023

A Practical Guide: Building a Data Mesh on Apache Kafka by Neil Avery

Neil is the CTO and Founder of Liquidlabs, a professional services company that specialise in streaming technologies. Previously he worked for Confluent in the Office of the CTO during their formative years, and has over 20 years of experience in distributed systems.

Summary:

Kafka is great, it’s un-opinionated and lets you build whatever you want; however many times we need opinions! We need some guidance on establishing the fundamentals of topic-naming & structuring, and schemas. We also need a consistent means of provisioning topics, handling their change, updating schemas, testing, knowing who owns which topics, as well as many other things.
SpecMesh OS brings all of this together under the banner of ‘data-mesh’. It provides a consistent means of modeling Kafka applications and resources as a ‘data product’, consistent tooling and automation to support governance and self-service that works with your tests and CI/CD. Think of it as an accelerator for doing things right when starting, building, and managing the ongoing lifecycle of your Kafka-based applications.

This talk deep dives into some of the common problems experienced on any Kafka project and how these problems should be avoided. Then it dives into Opensource automation and tooling that Specmesh uses to build enterprise level streaming applications. (specmesh.io)

Rethinking latency SLAs with Apache Kafka by Dare Famuyiwa

Dare is a Kafka expert and a solutions architect at Ultradovesystems. He has worked with a number of multinational organisations in helping them build their event-driven, data streaming platforms using modern, cloud native technologies powered by Kafka in distributed enterprise environments. He is a father of two who loves to ski, run, play football and dance.

Summary:

Latency and durability can mean different things to different businesses and Kafka practitioners. In this talk we are going to explore how delivering ultra-low latency for mission-critical applications has changed in the past few years, with the growing adoption of Apache Kafka. We will explore how Kafka can be used as a durable, event streaming platform for low latency workloads, zero downtime and no data loss, as opposed to P2P machine-to-machine connections.
The talk will deep dive into the various concepts of how to operate at near network speed and highlight the pro’s & con’s of each concept.
What you will learn from this talk;

1. How Kafka can operate at near-network speed
2. Trade-offs to consider between latency and durability, to keep Kafka performant compared to P2P microservices communication
3. An visual understanding of the durability guarantees of typical Kafka deployments – ZKafka, KRaft, Multinode Kafka cluster, SSL secured Kafka cluster, etc

Dare has explored this topic in more detail on his medium page

Note: Apologies for the lack of slide animation in the video. I’ll try to improve for next time. – Ian