In the rapidly evolving world of event streaming, organisations often struggle with the question of how to scale their Apache Kafka implementations beyond pilot projects and into enterprise-wide adoption. The challenge isn’t just technical—it’s organisational. How do you build capabilities across dozens of teams without creating bottlenecks or sacrificing quality?
In just 17 months, we transformed a telecommunications company in the UK from having a single Apache Kafka topic to supporting 58 mission-critical applications—all without building a massive central team. Their secret? A federated Event Driven Architecture (EDA) strategy combined with a robust enablement ecosystem.
Understanding the Federated Model for Apache Kafka Implementation
Traditional middleware implementations typically rely on centralised, shared-service models where a core team handles all development and integration work. While this approach ensures consistency and control, it quickly becomes a bottleneck as demand grows.
A federated EDA model takes a fundamentally different approach:
Federated Model Defined: A small core team of EDA experts who focus on empowering application teams rather than implementing everything themselves. The core team establishes patterns, provides training, and offers consulting support, but the actual implementation work is distributed across the organisation. As each team is educated in the correct way, this knowledge spreads throughout the organisation.
This approach paid dividends for the telco provider, enabling them to rapidly scale to 58 applications across business units (31 in their suppliers and logistics division and 11 in billing department) in less than two years. The federated model’s power comes from its multiplicative effect—each team you enable becomes another center of implementation capability.
Securing Executive Buy-In for Your EDA Strategy
Getting leadership commitment for a federated EDA strategy requires more than technical arguments. The telco company secured an £1 million investment by positioning EDA not as “just another integration tool” but as a fundamental transformation in how the business managed its data.
Beyond “Another Integration Tool”
The key was identifying specific business pain points that a real-time approach could solve. For instance, the company’s customers were calling customer service representatives to check broadband order status because data was only updated in batches every six hours.
The systems were a data-at-rest architecture with a lot of batch jobs, and that created a lot of problems – we needed to change the mindset of how they manage data within their walls.
Building the Business Case
The business case focused on how EDA would tangibly improve customer experience and operational efficiency. Senior leadership agreed to invest when they saw a clear strategy for using capital efficiently and making a meaningful difference in operations.
Naming and Framing for Success
Interestingly, the team deliberately avoided being known as “the Kafka team.” Instead, they positioned themselves as the “Event Driven Architecture team”—consultants who help business units understand their data needs and then recommend appropriate solutions.
Building Your Enablement Foundation
For a federated model to succeed, you need to build capabilities across the organization. First create a comprehensive learning ecosystem with both synchronous and asynchronous components.
Creating a Learning Ecosystem
The foundation of our approach was establishing a single source of truth for all resources. They used Confluence as their central hub, ensuring all training materials, documentation, event calendars, and contact information were easily accessible in one place.
Their content strategy balanced efficiency with effectiveness:
- Develop only what you must – They focused on creating custom content for company-specific implementations, security requirements, and architectural patterns.
- Leverage existing resources – For foundational Apache Kafka knowledge, they directed teams to Confluent’s excellent training resources and other external content.
- Create concise, focused content – They prioritized bite-sized learning modules that could be consumed independently or combined into larger learning paths.
This approach allowed us to create seven comprehensive learning paths catering to different roles, from business stakeholders who needed conceptual understanding to developers requiring hands-on technical knowledge.
The Lunch & Learn Approach
One particularly effective and low-cost initiative was their regular “Lunch & Learn” series—45-minute webinars featuring internal and external experts sharing knowledge about EDA implementation.
These sessions serve multiple purposes:
- Knowledge sharing from experts
- Showcasing internal success stories
- Building community around the technology
- Gathering feedback through polls and discussions
- Identifying future presenters and subject matter experts
Transitioning from Implementation to Consulting
As application teams gained proficiency, the core EDA team’s role evolved from hands-on implementation to higher-level consulting and innovation.
Evolution of the Core Team’s Role
Once you have started building those skill sets within the application teams, the core team can focus more on the higher-level stuff, instead of doing the transactional development tasks which you get at the beginning of each project. This evolution enabled them to take on beneficial use cases that weren’t part of funded projects—a key to continued innovation. You could even run a “Dragons Den” where developers could pitch ideas for solving business problems using EDA.
Measuring Progress Through KPIs
To demonstrate success, you should track several key performance indicators:
- Learning effectiveness: X% of learning path completers reported positive performance impact, with only X% “scrap learning” (content that wasn’t applied in practice).
- Demand and satisfaction: X% of participants desired additional learning paths, with a Net Promoter Score of X%.
- Ecosystem engagement: Nearly X monthly views of their Wiki / Confluence content—a X% increase year-over-year.
- Application growth: Tracking the number of applications implementing Apache Kafka across different business units.
These metrics will help justify continued investment and highlighted areas for improvement.
Practical Implementation Guide
If you’re considering a federated EDA approach, here are concrete steps to set yourself up for success:
Starting Your Federated Journey
- Begin with clear architectural patterns and governance guidelines
- Document your implementation standards
- Create reusable templates and code examples
- Establish clear security and compliance requirements
- Develop organisation-specific training content
- Focus on your specific environment and tools
- Create content addressing your unique security, networking, and risk posture
- Build role-specific learning paths that guide users through the necessary knowledge
- Make content accessible and bite-sized
- Use a central platform where all resources live
- Create concise modules that can be consumed independently
- Provide both introductory and advanced content to support different roles
Common Pitfalls to Avoid
- Underinvesting in organisational change management
- Changing how an organisation views and uses data requires intentional change management
- Without OCM, Apache Kafka risks becoming just “the new shiny object of the week”
- Failing to customise learning content
- Generic Apache Kafka training alone won’t enable teams to implement effectively in your environment
- Contextualising to your specific implementation is crucial for success
- Not continuously engaging your audience
- One-time training isn’t enough; continuous engagement maintains momentum
- Celebrate wins publicly to reinforce the value of the transformation
Conclusion
A federated approach to EDA implementation offers a compelling alternative to traditional shared service models, particularly for large organisations seeking to scale quickly. By investing in enablement rather than simply growing a central team, you can distribute capabilities across your organisation while maintaining governance and quality.
