Kafka Replication Explained: Replicas, ISR and min.insync.replicas

Replication is the reason a Kafka cluster can lose a broker without losing data. It is also one of the most misunderstood parts of Kafka. The defaults are not tuned for durability, and a few settings decide whether a broker failure is a non-event or an incident.

This guide explains how replication works, what the in-sync replica set really means, which settings prevent data loss and what happens when a broker fails. All configuration defaults below come from the Apache Kafka 4.4 broker configuration reference.

How does Kafka replication work?

Kafka replicates at the partition level. Each partition has one leader replica and zero or more follower replicas, each on a different broker. The replication factor is the total number of copies, so a replication factor of 3 means one leader and two followers.

Producers write to the leader, and consumers normally read from the leader too. Followers do not receive pushed data. Instead, each follower sends fetch requests to the leader, much like a consumer, and appends what it receives to its own copy of the log.

The leader tracks how far each follower has fetched. The highest offset that every in-sync replica has copied is called the high watermark. Consumers can only read records up to the high watermark, so they never see a record that could vanish if the leader failed.

Someone has to decide which replica leads. In a KRaft cluster that job belongs to the controller quorum, a small group of controller nodes that store the cluster metadata in a replicated log of their own. When a leader fails, the active controller picks a new leader from the in-sync replicas and tells the brokers. ZooKeeper is gone from Kafka 4.x, so KRaft is the only mode.

What is the in-sync replica set (ISR)?

The in-sync replica set, or ISR, is the list of replicas that are fully caught up with the leader. The leader is always in it. A follower stays in it as long as it keeps fetching and keeps up.

The rule is set by replica.lag.time.max.ms. If a follower has not sent a fetch request, or has not caught up to the leader’s log end offset, for at least this long, the leader removes it from the ISR. The default is 30,000 ms (30 seconds).

The ISR shrinks and grows on its own. A follower that falls behind because of a slow disk, a long garbage-collection pause or a network problem drops out. Once it catches up again, the leader adds it back. Frequent shrinking and expanding usually points to an overloaded broker or an unstable network rather than a Kafka bug.

The metric to watch is UnderReplicatedPartitions, which every broker reports. It counts partitions led by that broker where the ISR is smaller than the full replica list. In a healthy cluster it sits at zero. A value that stays above zero means you have less protection than you think.

Which replication settings prevent data loss?

Durability comes from a small group of settings working together. The table shows each one, its default and a sensible production value.

Here is how they fit together. With acks=all, every replica in the ISR must acknowledge a write before the producer treats it as successful. min.insync.replicas decides whether the ISR is large enough to accept the write at all. If the ISR is smaller than this number, the producer gets a NotEnoughReplicas error instead of a write that only one broker holds.

The Kafka documentation describes this exact pattern as the typical scenario: replication factor 3, min.insync.replicas of 2 and acks=all. A majority of replicas must persist each write. You can lose one broker and keep writing, and no acknowledged write lives on a single machine.

Two defaults deserve attention. default.replication.factor is 1, so auto-created topics have no copies at all. min.insync.replicas is also 1, which means acks=all can succeed with only the leader in the ISR. Set both explicitly rather than trusting the defaults. Our guide to producer settings covers the client side in more depth.

unclean.leader.election.enable should normally stay false. Turning it on lets an out-of-sync replica become leader as a last resort, which keeps the partition available but can silently drop records. Finally, set broker.rack on every broker so Kafka spreads each partition’s replicas across racks or availability zones.

Key takeaway: Replication factor 3, min.insync.replicas=2 and acks=all is the safe default; it protects brokers, not regions or bad deletes.

What happens when a broker fails?

When a broker stops, the controller notices that it has stopped sending heartbeats. For every partition that broker led, the controller elects a new leader from the remaining in-sync replicas. Producers and consumers refresh their metadata and carry on with the new leader, usually within seconds.

The failed broker also drops out of the ISR for every partition it followed. With replication factor 3 and min.insync.replicas=2, those partitions still have two in-sync replicas, so writes continue. If a second broker holding the same partition fails, the ISR falls to one. Producers using acks=all then receive NotEnoughReplicas errors until a replica returns. That is the cluster choosing safety over availability, which is what you asked for.

When the broker comes back, it removes any records that diverge from the current leader’s log, fetches what it missed and rejoins the ISR once it catches up. Recovery is automatic, but a large backlog can take time to replicate.

To see which partitions are exposed during or after an incident, use kafka-topics.sh:

kafka-topics.sh --bootstrap-server broker1:9092 --describe --under-replicated-partitions

kafka-topics.sh --bootstrap-server broker1:9092 --describe --under-min-isr-partitions

Both commands should return nothing on a healthy cluster. Kafka 4.x connects through --bootstrap-server; the old --zookeeper option no longer exists.

Is replication enough for disaster recovery?

No. Replication protects you against losing a broker, a disk or, with rack awareness, a single availability zone. It does not protect you against the failures that cause most real data loss.

Replication copies every change, including the bad ones. If someone deletes a topic, a misconfigured retention setting purges data or an application writes corrupt records, every replica faithfully copies the damage. Replication also lives inside one cluster, so losing a whole region or the cluster itself takes every copy with it.

For those cases you need protection outside the cluster:

• Cross-cluster replication with MirrorMaker 2 or a similar tool keeps a standby cluster in another region. Our write-up on multi-region Kafka covers the architecture.
• Backups give you point-in-time copies you can restore after a bad delete or corruption. See our guide to backing up a Kafka cluster.
• A tested recovery plan ties it together, with clear recovery time and recovery point targets. Our field-tested approach to disaster recovery for Apache Kafka walks through one.

Treat replication as the first layer of protection, not the whole plan. If you want a second opinion on your setup, our Kafka support team reviews replication and recovery designs for production clusters.

Frequently asked questions