At OSO we deploy self-hosted Apache Kafka for enterprise clients almost daily. The hard part at scale is not only brokers and partitions—it is keeping hundreds of application configurations consistent so a disaster-recovery failover or a promotion to production does not surprise you. This post explains how a GitOps management plane reduces that configuration drift.
Configuration drift is the gap between the state you think you have and the state that is running. On Kafka that usually means:
| Drift surface | What diverges | Typical symptom |
|---|---|---|
| Topics | Partition count, retention, cleanup policy, compression | App works in one region, fails or lags in another |
| Access | ACLs / RBAC for producers and consumers | Auth failures after failover |
| Schemas | Compatibility mode, subject versions | Consumers cannot deserialize after promote |
| Quotas / throughput | Produce/consume limits per client | Throttling only in production |
Enterprises often run many Kafka clusters per environment—multi-tenant brokers plus active-active replicas across regions—to isolate blast radius. That topology only helps if configuration stays aligned. Drift that sits unnoticed until a DR event is a reliability debt, not a paperwork issue. For the platform patterns that sit beside GitOps, see our self-service Kafka operations guide.
Picture a sandwich shop that starts with custom orders, then tries to recreate each sandwich from memory when demand explodes. Errors creep in. Kafka onboarding looks the same when every new team repeats the same manual steps:
Each repetition is a chance for a mistyped retention, a missing ACL or a schema left on the wrong subject. Multiplied across hundreds of applications, that is systematic configuration drift. Automation does not remove the need for good defaults—it makes the same good defaults land everywhere.
A GitOps management plane treats Git as the only write path for cluster state. Topic definitions, access rules and schema-related config live in YAML. A Kubernetes operator (or equivalent controller) applies that desired state so both sides of an active-active pair receive the same change.
When someone needs a new topic, they change the YAML in Git. The operator creates the topic with identical settings on each replicated cluster. The change is only considered complete when it has persisted on both; otherwise it rolls back. That transactional habit stops the classic failure mode: primary updated, replica forgotten, failover broken.
This is governance, not a substitute for broker monitoring. You still watch under-replicated partitions, consumer lag and disk. GitOps answers a different question: “Are we running the topic and ACL estate we intended?” Our Kafka Cruise Control guide covers rebalancing once the declared topology is correct.
Aligning replicas is only half the problem. Environments still diverge if engineers edit Prod directly.
We use one Git branch per environment. Branches are segmented and are not merged into each other:
| Branch | Who can change it | Purpose |
|---|---|---|
| Dev / Test | Application teams | Create and iterate on topic, ACL and schema config |
| Prod | Controlled promotion only | Holds the released snapshot for higher environments |
A Promote and Release automation snapshots the approved development configuration and pushes that release into the production branch. New work is always tested in development first; only then is a release created and promoted. If testing fails, you roll back with the normal Git history instead of hunting hand-made fixes on the brokers.
Useful extras in that promote step:
That pattern removes repetitive copy-paste operations and keeps Dev → Prod movement auditable.
Centralising configuration management for Kafka clusters tends to:
We have discussed open-sourcing parts of the Promote and Release automation. If you want help adopting a similar GitOps model on your platform, contact OSO or explore Kafka services and fully managed Kafka.
Key takeaway: Avoid Kafka configuration drift with GitOps: identical topic, ACL and schema changes from Git on every replica, one branch per environment, and promote-and-release with dry-run before production.
Configuration drift is when the running Kafka estate (topics, ACLs, schemas, quotas) no longer matches the intended or peer environment—often after manual changes on one cluster that never landed on its replica or in production.
Git holds the desired state. Controllers apply the same YAML to every cluster that should match, so humans are not the source of truth for each broker. Reviews and history live in Git instead of undocumented CLI sessions.
Separate branches stop Dev experiments merging straight into Prod. Promotion becomes an explicit release snapshot with checks, rather than a merge that carries half-finished topic definitions.
At minimum: a dry-run diff against the target environment, validation that required fields exist, and any environment-specific overlays (partitions, quotas). Prefer failing closed if a replica apply does not confirm.
No. GitOps governs declared configuration. You still need metrics and alerts for broker health, lag, disk and ISR. Use both: GitOps for consistency, monitoring for runtime truth.
Talk to OSO's Kafka engineers about GitOps topic, ACL and schema management across multi-cluster and multi-environment platforms.
CONTACT US