How OSO helped Hyperoptic move Kafka Connect to self-hosted Strimzi

Overview

Hyperoptic launched in 2011 as the UK’s first alternative full-fibre broadband network and finished 2025 with more than 440,000 customers. Behind that service sits a growing estate of network, customer and data platforms, and an engineering organisation that has chosen AWS as the place to run them.

Hyperoptic’s strategy is a multi-account AWS estate in which product teams serve themselves. Infrastructure comes from reusable Terraform modules, event-driven Java microservices come from standard templates, and Apache Kafka runs as a shared platform service. Since 2024, OSO engineers have worked inside Hyperoptic’s platform engineering team to make that model work in production, and today they contribute to its Platform and NetOps squads within Hyperoptic’s quarterly planning cycle.

This story covers three parts of that work: self-hosted Kafka Connect, real-time change data capture, and developer self-service. The strategic goals were clear. Hyperoptic needed a platform that could:

• Run every Kafka connector the same way in every environment, on infrastructure the platform team controls
• Replace batch ETL into the data warehouse with real-time change data capture through Kafka
• Let development squads create services, infrastructure and Kafka resources from approved templates
• Stay supported in production, with specialist Kafka engineers on hand when connectors misbehave
• Leave Hyperoptic’s own engineers able to own and extend all of it

This was less a single project than the groundwork for a platform that product teams can use without waiting for someone else to build it for them.

The Technical Challenge and Why They Engaged OSO Engineers

Hyperoptic’s Kafka brokers run on Amazon MSK, and its connectors originally ran on Amazon MSK Connect. The platform team wanted connectors to follow one deployment standard across every environment, with the same packaging, configuration and promotion path as the rest of the platform’s Kubernetes workloads.

The data team faced a related problem. Reporting depended on batch ETL into the existing data warehouse, with collection tied to single servers. A real-time pipeline meant capturing changes from the core database as they happened and landing events where the data team’s tooling could reach them. Old and new pipelines also had to run side by side, with evidence that they agreed, until the legacy warehouse could be retired.

Developers needed a better route to production too. Product squads were adopting Kubernetes and templated microservices, but every new service still needed a repository, a CI/CD pipeline, infrastructure, and Kafka topics, ACLs and schemas. Without a single self-service front door, that work queued behind the platform team.

All of it had to be delivered inside Hyperoptic’s controls: reviewed change, its security standards, and strict rules on where customer data may be processed.

Hyperoptic engaged OSO for its depth in Apache Kafka on both Amazon MSK and Kubernetes, its platform engineering experience on AWS, and its practice of building capability inside the client’s own team rather than around it.

The Solution

The OSO engineers joined Hyperoptic’s platform engineering team as embedded engineers. They worked in weekly iterations against a shared backlog aligned to Hyperoptic’s quarterly planning. Alongside delivery, they ran weekly knowledge-transfer sessions and a two-day Apache Kafka course for Hyperoptic’s developers.

The Results and Outcomes

Hyperoptic now runs Kafka Connect on infrastructure its platform team controls and streams database changes into its data platform as they happen. Its development squads have a self-service front door, and OSO engineers support the connector platform in production.

One way to run connectors, in every environment. Connectors run on self-hosted Kafka Connect managed by Strimzi, defined in Git and promoted through the same pipeline from development to production. More than 20 connectors now run this way.

Real-time change data in place of batch ETL. Debezium connectors feed database changes into Kafka as they happen, and S3 sink connectors deliver events to the data team’s Airflow, dbt and Iceberg tooling. This is the foundation for retiring the legacy warehouse.

A self-service front door for developers. Backstage templates let squads create a new service, with its repository, CI/CD pipeline and Kubernetes deployment, from an approved golden path instead of assembling it by hand.

Kafka resources by pull request. Topics, ACLs and schema registrations go through an automated, reviewed pipeline, which keeps the shared Kafka platform consistent as more squads build on it.

A platform team that owns its platform. Weekly knowledge-transfer sessions and a two-day Apache Kafka course mean Hyperoptic’s engineers run and extend these patterns themselves.

Supported in production by the engineers who built it. Under OSO’s managed service, connector incidents and upgrades go straight to Kafka engineers who already know the platform, with no first-line triage in between.

A long-term platform partnership. OSO engineers continue to work within Hyperoptic’s Platform and NetOps squads on core platform modernisation, security remediation and network diagnostics.

What OSO Delivered

• Moved Kafka connectors from Amazon MSK Connect to self-hosted Kafka Connect on Strimzi, with one deployment standard across environments
• Configured Debezium CDC and S3 sink connectors for the real-time data warehouse migration, with deployment automation and parallel-run validation
• Configured Backstage as the internal developer portal, with software templates, scaffolder integration with CI/CD and golden-path templates
• Productionised the pipeline for Kafka topics, ACLs and schema registration
• Supported the Strimzi-based Connect platform in production under OSO’s managed service
• Ran weekly knowledge-transfer sessions and a two-day Apache Kafka course for Hyperoptic’s developers