Building Bulletproof Disaster Recovery for Apache Kafka: A Field-Tested Architecture

Picture this: it’s 3 AM and your phone won’t stop buzzing. Your primary Kafka cluster, the one serving millions of transactions, has gone down, and the war room is already assembling. This isn’t hypothetical. The OSO engineering team has dealt with exactly this while operating Kafka platforms processing over 300 million transactions a day, with cross-datacenter replication peaks of 6.5 million messages per second.

The question isn’t whether disaster will strike your Kafka infrastructure, it’s when. Disaster recovery approaches that work for monolithic applications fall short on a distributed streaming platform like Apache Kafka. This guide sets out the framework OSO engineers use in production: failure modes, the economics, replication trade-offs, active-active and active-passive patterns, and a failover runbook built from real Kafka commands.

Updated October 2026: MirrorMaker 2 configuration, offset translation and failover commands have been checked against the Apache Kafka 4.3 geo-replication documentation and the MirrorMaker source. MirrorMaker 1 was removed in Kafka 4.0, so everything here uses MirrorMaker 2.

What Kafka failures need disaster recovery?

Kafka’s distributed design means failures show up differently than in traditional systems. Node-level failures, such as a broker crash, a failed disk or a JVM out-of-memory error, are common and Kafka’s built-in replication normally absorbs them. Cluster-level disasters are a different class of problem.

The most damaging failures take out several brokers at once. Imagine a replication factor of three and an unlucky sequence of events, perhaps a botched rolling upgrade or a rack-level power failure, that takes down all three brokers hosting the same partition replicas. That data is unavailable and producers start timing out. Network partitions and cascading failures triggered by small misconfigurations can be just as damaging.

Beyond technical failures are environmental disasters that no clever engineering prevents: floods, earthquakes, fires and long power grid outages. They demand geographic distribution of your Kafka infrastructure, which traditional DR planning often overlooks. OSO engineers have seen a lightning strike take down a datacenter’s entire power infrastructure, which is why geographic redundancy is essential rather than advisable.

How do you justify the cost of Kafka DR?

One of the hardest parts of Kafka DR isn’t technical, it’s financial. OSO engineers use a simple framework that works for both technical and business stakeholders: compare the cost of an outage with the cost of implementing DR.

The true cost of downtime goes well beyond lost revenue. Direct costs include lost transaction revenue (often millions in financial services), SLA penalties, regulatory fines, emergency overtime and infrastructure replacement. The hidden costs build up over time.

Customer attrition is often the biggest hidden cost. During a major outage at a financial services client, OSO engineers saw users not just switch to competitors temporarily but uninstall the application. Brand damage makes future customer acquisition more expensive, and lost market share can take years to win back.

In regulated industries, particularly financial services where OSO works, DR isn’t optional. Regulators commonly set maximum allowable downtime (often under four hours), data locality requirements, audit trails for failover procedures and regular DR testing. Penalties for non-compliance can dwarf the cost of doing DR properly.

Not every Kafka workload needs full DR. OSO engineers classify applications into three tiers:

Should Kafka DR use synchronous or asynchronous replication?

The choice between synchronous and asynchronous replication shapes your whole DR architecture, and production experience shows trade-offs that theory often misses.

With synchronous replication, one Kafka cluster stretches across datacenters, and producers using acks=all only receive an acknowledgment once every in-sync replica has the write. Consider a practical example:

Producer (Mumbai DC) -> one stretched Kafka cluster
                        |-- Brokers 1-3 (Mumbai DC,  broker.rack=mumbai)
                        |-- Brokers 4-6 (Delhi DC,   broker.rack=delhi)
                        |-- KRaft controller tie-breaker (third site)

Here a producer in Mumbai writes to a cluster with brokers in Mumbai and Delhi. Two details make this work. First, acks=all only guarantees a copy in both datacenters if replicas are placed rack-aware (set broker.rack per datacenter) and min.insync.replicas is high enough that the in-sync set must include the other site. Second, a KRaft controller quorum needs a majority to keep running, so with only two sites, losing the site holding most controllers stops the cluster. Place a controller in a third location as a tie-breaker.

Real-world numbers are sobering. Baseline latency inside a single datacenter is typically one to two milliseconds. With synchronous replication across datacenters about 100 km apart, OSO measured 10 to 20 milliseconds, and throughput dropped by 30 to 50% compared with a single datacenter.

OSO’s production data points to what we call the 100-kilometre rule: beyond that distance, synchronous replication becomes impractical. Network physics, not infrastructure quality, sets the limit, and no amount of tuning beats the speed of light.

Synchronous replication makes sense when transaction rates stay below about 10,000 per second, applications can tolerate latencies above 20 milliseconds, strong consistency is required (such as financial ledgers) and the datacenters are close together.

Asynchronous replication with MirrorMaker 2 decouples the source and target clusters, which allows global distribution:

Mumbai Kafka cluster -> MirrorMaker 2 -> Singapore Kafka cluster
(primary DC)            (replicator)     (DR DC)

Replication lag: 100 ms to 4 s depending on load

In OSO’s production environment, MirrorMaker 2 replicates from Mumbai to Singapore with steady-state lag of two to four seconds, handling peaks of 6.5 million messages per second and 700 megabytes per second. Asynchronous means a recovery point objective above zero: whatever hasn’t replicated when the primary fails is lost or delayed.

Storage economics matter too:

That doubling of storage feeds directly into infrastructure cost and has to be part of DR planning.

Should you run active-active or active-passive Kafka DR?

The choice between active-active and active-passive changes both operational complexity and resource use. For critical systems OSO prefers active-active, where both datacenters serve production traffic at the same time.

In a typical active-active deployment, OSO runs identical infrastructure in each datacenter:

Each site runs three MirrorMaker 2 nodes. Producers and consumers work in both locations, processing local traffic while replication keeps the other site up to date. A minimal dedicated-mode configuration for two-way replication looks like this:

clusters = mum, blr
mum.bootstrap.servers = kafka-mum:9092
blr.bootstrap.servers = kafka-blr:9092

mum->blr.enabled = true
mum->blr.topics = orders, payments
blr->mum.enabled = true
blr->mum.topics = orders, payments

Topic naming is crucial in active-active. There are two approaches.

1. Prefixed replication (the default):

Mumbai: orders    -> Bangalore: mum.orders
Bangalore: orders -> Mumbai:    blr.orders

MirrorMaker 2’s default replication policy prefixes each remote topic with the source cluster alias. That gives clear lineage and automatic loop prevention, because MirrorMaker won’t replicate a topic back to the cluster it came from. The cost is that consumers who need a global view subscribe to several topics, for example with a pattern like .*orders.

2. Identity replication:

Mumbai: orders <-> Bangalore: orders

IdentityReplicationPolicy keeps the same topic names in both datacenters, which simplifies consumers. But the Kafka source is explicit that MirrorMaker cannot prevent cycles with this policy, so the replication topology must be acyclic. In practice that makes identity replication a good fit for one-way active-passive DR and migrations, not two-way active-active.

If you must run two-way replication with identical names, you need your own loop guard. One approach is for producers to stamp an origin header and for MirrorMaker to drop records that already carry it, using a Connect filter transform with a header predicate (KIP-585):

blr->mum.transforms = dropOwn
blr->mum.transforms.dropOwn.type = org.apache.kafka.connect.transforms.Filter
blr->mum.transforms.dropOwn.predicate = fromMumbai
blr->mum.predicates = fromMumbai
blr->mum.predicates.fromMumbai.type = org.apache.kafka.connect.transforms.predicates.HasHeaderKey
blr->mum.predicates.fromMumbai.name = x-origin-dc-mumbai

HasHeaderKey matches on header name only, so each site needs its own header key. Test this carefully, because a missed header means an infinite loop. For most teams the default prefixed policy is the safer route to active-active.

Active-passive reduces operational complexity at the cost of resource efficiency. The passive datacenter sits idle in normal operation and takes over only when the primary fails. It suits organisations with limited Kafka operations experience, recovery time objectives under five minutes, applications needing session affinity, or data residency requirements.

How do you size and tune MirrorMaker 2?

Sizing starts from partition counts and available cores. OSO’s field rule of thumb is to plan about one core per replicated partition at peak, then add 50% headroom for rebalancing and catch-up:

MirrorMaker nodes (baseline) = source partitions / cores per node
Recommended nodes            = baseline x 1.5 (headroom for rebalancing)

Example:
- Source partitions: 1,000
- Cores per MirrorMaker node: 40
- Baseline: 1,000 / 40 = 25 nodes
- Recommended: 38 nodes (50% headroom)

Treat that as a starting point and load test with your own message sizes and compression. Tuning makes a big difference. In MirrorMaker 2 dedicated mode, client settings take a cluster prefix: the consumer reading the source uses mum.consumer., and the producer writing to the target uses blr.producer.:

tasks.max = 40

mum.consumer.fetch.min.bytes = 100000
mum.consumer.isolation.level = read_committed

blr.producer.batch.size = 100000
blr.producer.linger.ms = 100
blr.producer.compression.type = lz4
blr.producer.acks = all
blr.producer.enable.idempotence = true
blr.producer.max.in.flight.requests.per.connection = 5

Older versions of this guide recommended acks=1 and retries=3. For DR, keep acks=all with idempotence (the Connect default) and leave retries at the default, otherwise a broker failover on the DR side can silently drop replicated records. From Kafka 3.5, MirrorMaker 2 can also replicate exactly-once in dedicated mode:

blr.exactly.once.source.support = enabled
dedicated.mode.enable.internal.rest = true

How do you monitor Kafka DR replication?

Monitoring tells you whether you can actually fail over. OSO tracks four metric groups. MirrorMaker 2 exposes the replication metrics through JMX under kafka.connect.mirror, including replication-latency-ms and record-age-ms for the source connector and checkpoint-latency-ms for checkpoints.

These give early warning of replication problems before they hurt your ability to recover.

What is the Kafka failover runbook?

Failover needs careful planning and regular practice. This is OSO’s sequence, using the standard Kafka command-line tools (.sh scripts in the Apache distribution).

1. Verify the primary has failed

kafka-broker-api-versions.sh --bootstrap-server kafka-mum:9092 --command-config admin.properties

kafka-metadata-quorum.sh --bootstrap-server kafka-mum:9092 --command-config admin.properties describe --status

ping -c 10 kafka-mum-broker-1.mumbai.internal

Confirm with your alerting (for example Prometheus and Alertmanager) that this isn’t a false alarm before you commit to failover.

2. Fail over producers

sed -i 's/kafka-mum:9092/kafka-blr:9092/g' producer.properties
systemctl restart kafka-producer-service

3. Move consumers with translated offsets

Offsets in the DR cluster don’t match the primary’s, so consumers need translated offsets. MirrorMaker 2’s checkpoint connector records them (every 60 seconds by default). If you set sync.group.offsets.enabled = true on the replication flow, MirrorMaker also writes translated offsets straight into the DR cluster’s consumer groups whenever no consumers in that group are active there, so consumers can start with the same group.id.

kafka-consumer-groups.sh --bootstrap-server kafka-blr:9092 --describe --group myapp

If you translate offsets yourself (for example with the RemoteClusterUtils.translateOffsets API), write them as CSV rows of topic,partition,offset and apply them. --reset-offsets does nothing without --execute:

kafka-consumer-groups.sh --bootstrap-server kafka-blr:9092 --group myapp \
  --reset-offsets --from-file translated-offsets.csv --dry-run
kafka-consumer-groups.sh --bootstrap-server kafka-blr:9092 --group myapp \
  --reset-offsets --from-file translated-offsets.csv --execute

4. Validate the data

kafka-get-offsets.sh --bootstrap-server kafka-blr:9092 --topic mum.orders --time -1

kafka-console-consumer.sh --bootstrap-server kafka-blr:9092 \
  --topic mum.orders --from-beginning --timeout-ms 60000 \
  --property print.key=true --property print.value=false \
  | sort | uniq -d | wc -l

Some duplicates are expected with at-least-once replication, so consumers should be idempotent. A systematic runbook keeps data loss small and recovery fast during a real disaster.

Testing is what makes DR real. OSO runs monthly tests of single broker failures, network partitions and rolling upgrades. Quarterly tests escalate to full datacenter failover, multi-region failback and cross-team coordination. Annual unannounced failover drills test the whole organisation, including executive stakeholders.

Kafka disaster recovery takeaways

Bulletproof disaster recovery for Apache Kafka takes more than knowing the technology. It combines architectural patterns, economic analysis and operational discipline. The best DR strategy is one that’s regularly tested, economically justified and architecturally sound.

Key takeaway: choose synchronous stretch clusters only within about 100 km, otherwise replicate asynchronously with MirrorMaker 2, sync consumer offsets and rehearse failover until the runbook is routine.

Start by assessing your current Kafka deployment against these patterns. If your primary Kafka cluster disappeared right now, how long would recovery take and what would it cost? If the answer makes you uncomfortable, it’s time to act.

Related reading:

• How to back up Kafka Schema Registry and restore it
• Best managed Kafka services

Frequently asked questions