Kafka: Data Mesh and Kafka Streams | OSO

In these presentations, you will delve into the world of Apache Kafka and explore how to build a robust Data Mesh architecture while implementing a Kafka Streams microservices framework. Learn the essential steps and strategies to leverage the power of Kafka Streams for scalable data processing, enabling real-time streaming applications and data-driven insights. Discover the key components and best practices to design, develop, and deploy Kafka Streams microservices, enabling seamless data integration and efficient stream processing. Whether you’re a beginner or an experienced developer, this guide will equip you with the knowledge and tools to harness the full potential of Kafka Streams in building a resilient and scalable data infrastructure.

If you had the opportunity to attend our incredible meet-up in April, we want to extend a huge thank you! We genuinely hope you had an amazing time, just like we did!

If, unfortunately, you couldn’t make it or if you’re eager to relive the experience, we’ve got you covered! You can now enjoy the recordings of both talks below. Don’t miss out! Also, our last meeting 

Summary: 

This talk gives an insight into the challenges faced & lessons learnt at Funding Circle as we built a distributed loan state management, money movement and accounting platform with Kafka Streams. It will cover managing multiple event streams, ordering issues and how to think about corrections, exceptions and sensitive information in an immutable data platform.

Speaker: Matt Searle – Director of Technology, Solutions & Architecture at Funding Circle

Matt has been a distributed systems engineer since 1997, working with Apache Kafka and Kafka Streams since 2017 (v0.8) as a Principal Engineer / Distributed Systems Architect. He’s built fully Kafka based financial systems and platforms with Kafka Connect, Avro Schemas and Kafka Streams, using JVM/Clojure, Ruby/Python, and even a little bit of C. His current focus is on looking at how to best leverage serverless (AWS) integrations and kSQLdb across a Kafka based event sourced system.

Summary: 

 A talk of two halves. The first will focus on the challenges encountered, and solutions found, of building a scalable microservice architecture on top of Kafka, covering the architecture and processes transitioning from an uncoordinated approach, to a highly scalable, reliable, observable and testable Kafka Streams microservice architecture. The second half is a demonstration, covering some of the solutions uncovered and discovered on that journey. Combined, these provide a streamlined and accelerated development & testing process that may be of interest to others. So much so, we’ve sourced them at https://www.creekservice.org.

Key talk takeaways:

• Decomposition of complex microservice architecture into distinct functional areas, with defined APIs, using the domain-driven-design concept of aggregates a.k.a. bounded-contexts.
• Utilising GitHub’s template repositories to accelerate development.
• Quick and easy YAML based functional testing of a microservice’s, or services’, business logic.
• Capture code coverage metrics from functional testing of services running in Docker containers.
• Debugging of services running in Docker containers.

Speaker: Andrew Coates – Former Confluent & Apple Engineer

Andrew describes himself as an engineer with a love for understanding & solving problems, building things and getting sh*t done. He has a passion for developing supportable, event driven, scalable architectures. He’s built, supported, and managed large scale Kafka clusters at Apple; worked in the ‘streams team’ at Confluent on KsqlDb and Kafka Streams; and recently headed up the Kafka team in a stealth financial startup, building a microservice architecture on top of Confluent Cloud’s Kafka offering.