Building Multi-Region Orchestration with Apache Kafka: A Pull-Based Architecture

Updated October 2026: We have reworked this guide into a question-led format, added a comparison of push and pull orchestration, a short FAQ and links to related OSO guides. The architecture itself comes from a production multi-region workflow platform that OSO engineers worked on, and it remains relevant on Kafka 4.x, which runs entirely without ZooKeeper and makes the new consumer group protocol (KIP-848) generally available.

Many teams start workflow orchestration with HTTP: an orchestrator calls service A, waits, then calls service B. That works for a handful of services in one region. It breaks down when workflows run for hours, span dozens of services and must keep customer data inside specific regions. This guide explains how a pull-based Kafka workflow orchestration architecture solves those problems and what to watch for if you build one.

Why does HTTP-based orchestration fall short at scale?

HTTP orchestration falls short because it couples applications to service locations, mixes workflow logic with infrastructure concerns and pushes back-pressure handling onto every caller.

Tight coupling between applications and services. User-facing applications need explicit knowledge of backend endpoints, load balancers and network locations. Any refactoring, such as splitting a monolith or moving a service to a new region, means updating every caller. Routine architectural improvements turn into coordination exercises across several teams and release cycles.

The kitchen-sink workflow. Workflow definitions end up holding both the logical structure (which steps run, in which order) and the implementation details (which service to call, timeouts, retries and circuit breakers). Adding a validation step means editing code that also manages HTTP failure handling.

Back pressure and rate limiting. In a push model, when service B is overloaded, service A must handle 429 Too Many Requests responses, implement exponential backoff, keep a local queue of failed requests and shed load to avoid cascading failures. That complexity repeats for every service-to-service interaction.

Static service registration. Traditional orchestrators need the network location of each service. Adding capacity in a new region or running a canary means changing orchestrator configuration first, so the service registry becomes a bottleneck.

Regional data locality. When customer data must stay in a region for GDPR, data residency laws or financial regulation, HTTP orchestration pushes region-aware routing into every application. Duplicated logic increases the risk of a compliance mistake.

Redeployment brittleness. Without dynamic health awareness, an orchestrator cannot pause work, let a service drain, redeploy it and resume. Teams must choose between interrupting running workflows and delaying an urgent patch.

How does pull-based Kafka orchestration work?

Pull-based Kafka orchestration inverts the control flow: instead of the orchestrator pushing commands over HTTP, services pull work from Kafka topics when they have capacity.

Inverting the control flow. A service under heavy load simply consumes more slowly. The orchestrator does not need retry logic or rate limits, because unprocessed work waits safely in Kafka’s durable log until a consumer is ready for it.

Decoupling applications from implementations. Applications describe workflows as recipes that reference logical steps such as validate_customer_data. They never name the service or region that will run a step. Any service that has registered that capability can pick the work up, so services can be refactored, redeployed or moved without touching application code.

Topic-based communication. Request topics carry commands from the control plane to services, and response topics carry results back. The orchestrator publishes a step request, a subscribed service processes it and publishes the result, and the orchestrator decides the next step. No service ever needs the network address of another.

Why do Kafka’s guarantees matter for orchestration?

Kafka’s guarantees matter because a workflow engine must never lose, duplicate or reorder a step.

• Durable storage: commands and state transitions persist through outages. A service that is offline during a deployment resumes from its last committed offset.
• Exactly-once processing: for non-idempotent steps such as charging a customer £1,000, Kafka transactions stop a crashed consumer from reprocessing a command it already acted on.
• Ordering within partitions: partitioning topics by workflow execution ID keeps every step of one execution in order, even with thousands of executions running in parallel.

What are the core components of a Kafka orchestrator?

A Kafka orchestrator has three core components: a recipe abstraction, a dynamic shard registry and a regional control plane.

Recipe abstraction. Recipes are declarative workflow definitions, expressed in YAML or a Kotlin DSL in the platform we describe. They support conditional branches, loops over items, parallel processing and worker pools. A data migration recipe might run pre-checks, conditionally migrate users, set up the destination, run ETL jobs in parallel per partition, upload attachments and validate the result, without saying which service runs any step.

Dynamic shard registry. A real-time registry records which service shards are available in which regions, which logic identifiers they can execute and how healthy they are. When a service starts, it declares something like “I am service Alpha in EU-West and I can execute these five steps”, then sends heartbeats. If a dependency such as a database slows down, it reports degraded health and the orchestrator routes work elsewhere.

Control plane. A control plane runs in each region and manages recipe executions, step executions and buckets with Kafka Streams. When a workflow started in region A needs a step that touches data held in region B, the region A control plane coordinates with region B through the message relay, so the data never leaves its region.

Why use SDKs instead of HTTP APIs for Kafka workflow orchestration?

SDKs give services type-safe contracts and hide the distributed-systems plumbing, at the cost of maintaining one SDK per language.

Type safety. With Java and Kotlin SDKs, step executors implement typed interfaces. If a workflow adds a required field, services that have not been updated fail to compile rather than failing in production.

A simpler integration path. Services add the SDK, implement executor interfaces (a short command executor for quick operations and a long command executor for multi-hour work) and register their capabilities. The SDK handles consumers, producers, heartbeats and the orchestrator protocol, so a developer writes a method that takes a request and returns a response.

Hidden distribution complexity. For long-running work, the instance that receives a request may not be the one that finishes it. The SDK coordinates consumer groups and offsets so the response still reaches the orchestrator.

The trade-offs are real:

• Language lock-in: each language needs its own SDK. Starting with JVM languages made sense because they dominated the target estate, but adding Go, TypeScript or Python means rebuilding consumer management, heartbeats and registration each time.
• Platform coupling: services depend on the SDK’s release cycle, much like any framework upgrade.
• Dual interfaces: SDKs suit internal services. External clients and third parties still need HTTP APIs, so some services support both.

How does Kafka Streams manage workflow state?

Kafka Streams manages workflow state through event sourcing: each entity is the result of applying its change events in order, held in local state stores backed by changelog topics.

Entity lifecycle. The control plane models RecipeExecution, StepExecution and Bucket entities. A RecipeCreated event creates an execution with status Running, StepCompleted events update progress and a RecipeCompleted event moves it to Success. Any workflow’s current state can be rebuilt by replaying its events.

State stores. RocksDB-backed state stores give low-latency key-value access. The aggregate operator handles the fetch, modify and persist cycle, while Kafka Streams handles replication through changelog topics. If you are weighing Kafka Streams against other engines for this kind of stateful work, see our Kafka Streams vs Flink comparison.

Topology design. Topologies follow domain boundaries: one for recipe executions, one for step executions and one for buckets. Each groups events by key, aggregates state transitions, emits domain events and scales independently.

Buckets for loops. When a loop processes many items, such as 10,000 customer records, a Bucket entity acts as a map of queues keyed by step name. Fast steps keep producing while slower downstream steps catch up, so loop iterations run in parallel instead of in lockstep.

References instead of payloads. Compacted topics have message size limits (1 MB by default), so buckets store item references and the data itself lives in separate state stores.

Persistent stores for rebalancing. Rebuilding large in-memory stores from changelogs during a rebalance pauses processing. Persistent RocksDB stores let an instance recover most state from local disk and replay only recent changelog entries. Kafka 4.0’s new consumer group protocol also reduces stop-the-world rebalances for plain consumers; our guide to the new consumer rebalance protocol (KIP-848) explains how it works.

How do you route messages between Kafka clusters in multiple regions?

For multi-region Kafka orchestration, a header-based message relay routes individual command messages between clusters, rather than mirroring whole topics.

Why not MirrorMaker? MirrorMaker is built for high-throughput replication of entire topics. Orchestration sends small command messages that must be routed dynamically to one target cluster, so topic-level replication moves far more data than needed. For whole-cluster replication and failover, MirrorMaker remains the right tool, as covered in our Kafka disaster recovery architecture guide.

Header-based routing. Each message carries its target cluster and target topic in Kafka headers, like an address on an envelope. A control plane in region A publishes to a local topic with headers pointing at region B, and the relay delivers it.

Cut and paste semantics. The relay “cuts” a message from a source topic and “pastes” it into the target topic. Only messages that must cross regions incur transfer costs; most steps run in the same region as their control plane and never leave it.

Kafka Connect implementation. The team built custom source and sink connectors. The source connector reads from source clusters, inspects headers and sorts messages into per-region topics in a local relayer cluster. Sink connectors in each region read only their region’s topic and produce to the target cluster.

Produce locally, consume remotely. Producer-to-broker traffic involves more round trips than consumer-to-broker traffic, so the relay always produces to a nearby cluster and consumes across the network. Only one hop crosses regions.

Exactly-once and ordering. Interleaved messages for the same target must arrive in order or workflow state is corrupted. Getting exactly-once delivery across cluster boundaries meant careful tuning of Kafka Connect’s transactional settings and the connectors’ batching. If you need the same consistency between a database and Kafka, the outbox pattern solves a related problem.

When should you choose Kafka-based orchestration?

Choose Kafka-based orchestration when workflows are long-running, span regions with data residency rules, or involve services that change often.

• Multi-region stateful services: the shard registry and regional control planes route steps to services in the right region without a complex HTTP service mesh.
• Long-running workflows with 100+ steps: work survives redeployments because it waits in Kafka topics until the service is back.
• Frequent service refactoring: recipes reference logical steps, so service teams can reorganise backends without changing application workflows.

Key takeaway: Pull-based Kafka orchestration works because services pull work when they are ready. Kafka’s durable, ordered log then gives you back pressure, safe redeployments and regional routing without extra machinery in every service.

For a simple workflow inside one region with a few services, a conventional orchestrator or plain HTTP is usually enough. The Kafka approach pays off when coupling, scale and compliance start to hurt.

What operational lessons apply to Kafka orchestration?

The main operational lessons are to separate control and data planes, make discovery dynamic, keep state stores persistent and keep messages small.

Separate the control plane from the data plane. Keep orchestration logic apart from the business logic that touches customer data. Control planes can then run globally while data processing stays inside its region.

Invest in dynamic discovery and health. Services should self-register capabilities, regions served and health, using Kafka topics so every control plane sees the same view. “Alive” is not enough; services should say whether they are healthy and ready for full load or degraded.

Weigh SDKs against HTTP. A polyglot estate may get better reach from HTTP first. A mostly JVM estate gains a lot from typed SDKs, which reduced integration bugs and sped up service onboarding in this project.

Keep state stores persistent. In testing, large in-memory stores took five to ten minutes to rebuild during rebalances, stalling workflows. Persistent stores cut that to under thirty seconds.

Manage message size. Store references in entities and large data in separate stores. Kafka is built for high-throughput messaging, not for moving megabyte-sized blobs.

Cost cross-region traffic carefully. The team estimated that replicating whole topics with MirrorMaker would have cost around six times more than the purpose-built relay for this command-and-control traffic.

Across the platform, moving from HTTP push to Kafka pull removed whole classes of problems (back pressure, discovery brittleness and tight coupling) and roughly doubled throughput on real workloads.

If you are designing a multi-region Kafka platform or want a review of an existing one, OSO’s Kafka consulting services and Kafka support teams can help.

Frequently asked questions