How to Back Up Your Kafka Schema Registry (And Why Your Current Approach Probably Won’t Restore)

Most Kafka teams have something they call a Schema Registry backup: a curl script in a cron job, a console consumer dump, an export from a governance tool. Very few have ever tested whether it actually restores. And on the day they need it, a pattern shows up in the forum threads: schemas restored, IDs reassigned, messages unreadable, and a final post that reads “Finally, we delete all unprocessed data and start over.”

That outcome is more common than the Kafka community likes to admit. The OSO engineers have seen several production incidents where teams with otherwise solid disaster recovery discovered, mid-incident, that their Schema Registry recovery story was incomplete. It’s why we built kafka-backup, an open-source point-in-time recovery tool for Apache Kafka, with Schema Registry backup and restore as an enterprise feature. In one well-documented case on a public Kafka forum, an enterprise on managed cloud infrastructure lost its Kafka nodes along with Schema Registry. When everything came back, the schema data was gone, stream processing applications failed because they couldn’t resolve schema IDs, and there was no supported way to recreate schemas with their original IDs. Their last option was to delete unprocessed data and start over.

The lesson is simple, if uncomfortable: backing up Schema Registry is not the same as backing up schema definitions. A credible backup captures the schemas, their compatibility settings and cross-references, and preserves the schema IDs embedded in every Kafka message on disk. This guide covers what to capture, how to capture it and how to restore it without breaking consumers.

Updated October 2026: REST endpoints and IMPORT mode behaviour have been checked against the Confluent Schema Registry API reference, and the worked example updated for kafka-backup Enterprise 0.6.1.

Why does the wire format make Schema Registry backup hard?

Every Kafka message serialised with Avro, JSON Schema or Protobuf in the Confluent wire format starts with a five-byte prefix. The first byte is a magic byte, always 0x00. The next four bytes are a big-endian integer holding the schema ID. Then comes the payload.

The schema itself, its field names, types, namespaces and defaults, is not in the message. Only a pointer is.

When a consumer receives a message, it reads the magic byte, extracts the schema ID, asks Schema Registry for that ID, fetches the schema and only then deserialises the payload. Break any link in that chain and the consumer doesn’t degrade gracefully. It can’t read the message at all.

That’s why the OSO engineers call Schema Registry a “silent dependency”. It works reliably day after day, so teams forget it’s in the critical path. Nothing reminds you it exists, and the first time you find out your backup is broken is during an incident, the worst moment to improvise.

The implication is concrete: backing up schema definitions isn’t enough. The IDs matter, because every message on disk references them. In one public GitHub issue on the open-source Schema Registry project, a user reported that after a restart schema ID 221 simply wasn’t there. The error was Schema not found with error code 40403, and their stream processing applications stopped. The definition probably still existed in source control, but the binding between ID 221 and that definition was gone, and millions of messages were unreadable.

Schema content can usually be reconstructed. The IDs embedded in millions of Kafka messages can’t. That asymmetry is what catches teams out.

Why do common Schema Registry backup approaches fall short?

Before rolling your own, it helps to understand why the obvious approaches struggle in production.

The console consumer dump

The most common suggestion is to dump the internal _schemas topic with kafka-console-consumer. The records are readable JSON, but their format is an internal implementation detail rather than a stable public contract, so an upgrade can quietly change what your backup contains. You can’t filter by subject, restoring means replaying records into a new _schemas topic in exactly the right order, and it doesn’t fit the rest of your backup pipeline. It’s also unavailable on managed services where you can’t read internal topics.

The curl script

A better approach uses the Schema Registry REST API. There’s a well-known case of an engineer who extracted schemas this way before the registry process restarted and its in-memory state was lost. There were no replicas and the _schemas topic was gone, but Schema Registry was still serving from memory, so they had a narrow window to curl every schema. It worked, barely, and they restored into an empty registry with IMPORT mode. But curl scripts are fragile: no error handling, retries or rate limiting, no authentication beyond what you bolt on, no understanding of schema references, and no answer for restoring original IDs into a populated target.

Continuous replication tools

Vendor-native options replicate continuously between registries, but they need specific platform versions and enterprise licences, and they’re built for replication, not point-in-time backup. They don’t give you a snapshot matching a point-in-time Kafka data backup. Public forum threads describe managed Kafka users finding that schema data was not preserved during a failover, leaving them to recover schemas and data as separate manual steps in the middle of an incident.

Governance platforms

Schema governance tools can back up schemas, but as a separate artefact from your Kafka data. You end up with two backups and two restore processes, and you must make sure schemas are restored before data. That coordination falls on you, and coordination under pressure is exactly where incidents go wrong.

The common thread: none of these keep schemas in the same artefact as your Kafka data, and most can’t restore schemas with their original IDs.

What must a complete Schema Registry backup capture?

A backup that will actually restore needs four things.

1. Every schema version, through the REST API

The REST API beats a _schemas dump because it’s a stable public contract, works on self-hosted and cloud registries, returns inspectable JSON and supports subject filtering. Walk it: GET /subjects for the list, GET /subjects/{subject}/versions for each subject, then GET /subjects/{subject}/versions/{version} for each definition. The response includes the schema string, schema type (Avro, JSON Schema or Protobuf), global schema ID and references.

2. Compatibility configuration

Capture the global compatibility level with GET /config and per-subject overrides with GET /config/{subject}. Without them, the restored registry may reject schemas producers expect to register, or accept ones it should reject. Compatibility is part of the contract between producers and consumers, and losing it silently changes that contract.

3. Mode state

Capture the registry mode with the /mode endpoints, globally and per subject. IMPORT mode is what makes ID-preserving restore possible, and you need to know which mode to return each subject to afterwards.

4. Schema references and dependency order

The references field matters for Protobuf and any schema with cross-references. Say orders-value references a Money type in common-money-value and an Address type in common-address-value. You can’t restore orders-value first, because Schema Registry rejects it until the referenced schemas exist. The OSO engineers build a directed acyclic graph of dependencies and topologically sort it so referenced schemas are restored first. kafka-backup records that order in the backup manifest, so the restore follows it instead of recomputing it, and genuinely unsatisfiable dependency cycles are rejected.

How do you restore Schema Registry without breaking consumers?

Schema Registry normally assigns IDs itself; you don’t choose them. If the source has orders-value v1 at ID 1 and you restore into a registry that already holds other schemas, it may get ID 47. Your messages still say 0x00 0x00 0x00 0x00 0x01, so consumers fail.

Two strategies work.

Strategy 1: Force original IDs with IMPORT mode

Schema Registry has an IMPORT mode that accepts explicit IDs and version numbers. Normally POST /subjects/{subject}/versions hands out the next free ID. In IMPORT mode, you include id and version in the request body and the registry honours them.

curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"mode": "IMPORT"}' https://dr-schema-registry:8081/mode/orders-value

curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data @orders-value-v1.json \
  https://dr-schema-registry:8081/subjects/orders-value/versions

curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"mode": "READWRITE"}' https://dr-schema-registry:8081/mode/orders-value

Here orders-value-v1.json holds the schema, schemaType, references, id and version from the backup. Repeat for every subject in dependency order and the target ends up with the same IDs as the source. Two prerequisites: the registry must start with mode.mutability=true in its properties file, or mode changes are rejected, and by default Schema Registry refuses to switch to IMPORT once schemas exist unless you add ?force=true. Forcing IMPORT on a populated registry risks ID collisions, so this strategy is right when the target is empty or you know its IDs don’t conflict.

Strategy 2: Rewrite the schema IDs in each message

When you can’t control the target’s ID space, such as a shared multi-team registry or a different registry product, let the target assign whatever IDs it wants. As each schema registers, record the mapping, for example source ID 5 became target ID 23, and build a complete translation table. Then, during the Kafka data restore, which always runs after the schema restore, process each record as it’s written to the target cluster: read the magic byte and the four-byte ID, look up the target ID and replace those four bytes if they differ. The payload is untouched and the backup itself isn’t modified, so consumers on the target see IDs that resolve against the target registry.

Checking for a 0x00 magic byte is a useful guard, but on its own it’s a heuristic: a raw or custom payload can also start with 0x00. That’s why production tooling should be told explicitly which topics, keys and values use which serializer, rather than guessing from the first byte.

How does OSO kafka-backup handle Schema Registry backup?

The OSO engineers built kafka-backup to close this gap. Schemas are captured alongside Kafka topic data in a single artefact, so there’s no coordination between separate tools. Add an enterprise.schema_registry section to your backup configuration (configuration reference):

mode: backup
backup_id: daily-2026-04-06
source:
  bootstrap_servers: ["kafka:9092"]
storage:
  backend: s3
  bucket: kafka-backups

enterprise:
  schema_registry:
    url: "https://schema-registry:8081"
    auth:
      type: basic
      username: ${SR_USERNAME}
      password: ${SR_PASSWORD}

One command captures topics and schemas together, with no second tool or schedule (add --schema-only for a registry-only snapshot):

kafka-backup-enterprise backup --config backup-config.yaml

The output is human-readable. Schemas sit in a schema-registry/ directory inside the backup prefix, organised by subject:

{backup_id}/
  manifest.json
  topics/...
  schema-registry/
    _manifest.json
    _global_config.json
    subjects/
      orders-value/
        _metadata.json
        v1.json
        v2.json
      payments-value/
        _metadata.json
        v1.json

Each version file holds the full schema, type, global ID and references. The schema manifest records subjects, counts and dependency order. You can inspect the whole artefact straight from S3 or your storage backend, which helps when you need to know exactly what’s in a backup before committing to a restore.

For a disaster recovery restore into a fresh registry, preserving original IDs is the default. Each replayed topic declares how its keys and values are serialised:

mode: restore
backup_id: orders-daily
storage:
  backend: filesystem
  path: /var/backups/kafka
target:
  bootstrap_servers: ["dr-kafka:9092"]
  topics:
    include: ["orders"]
enterprise:
  schema_registry:
    url: https://dr-schema-registry:8081
    auth:
      type: basic
      username: ${SR_USERNAME}
      password: ${SR_PASSWORD}
  registry_restore:
    source: confluent
    destination: confluent
    id_policy: preserve
    serializers:
      orders:
        key:
          source: { type: raw }
          target: { type: raw }
        value:
          source: { type: confluent, namespace: "" }
          target: { type: confluent, namespace: "" }

kafka-backup-enterprise validate-restore --config restore.yaml --format json
kafka-backup-enterprise restore --config restore.yaml

For a migration where the ID space is out of your hands, set id_policy: rewrite: the destination allocates IDs and kafka-backup rewrites record framing during replay using the explicit serializer profiles. In Enterprise 0.6.1 the released restore workflow covers self-hosted Confluent Schema Registry and Apicurio Registry v2 and v3; Confluent Cloud registry restore is outside that release’s scope, so check the restore guide for current support.

The restore options are deliberately conservative. Existing entries on the target are reused only after their schema and references are verified as equivalent, destructive overwrite is rejected, and preserving IDs can’t be combined with rewriting them. ID preservation uses subject-level IMPORT mode, never a forced global IMPORT, and temporary subject modes are put back afterwards. Conflicts, missing dependencies or unknown record IDs stop the restore before any Kafka data is replayed.

The operational details that turn a script into infrastructure are built in. A rate limiter (25 requests per second by default) keeps backups clear of API limits, with retries and backoff on retryable responses. Authentication supports basic auth, mTLS and no auth, with credentials read from environment variables so secrets stay out of config files. validate-restore and dry-run mode check registry conflicts, dependencies and record framing without changing anything, so you see problems before a real restore.

Schema Registry backup takeaways

Schema Registry backup sounds boring until the day you need it, when it becomes the most important thing in your infrastructure. The mechanics aren’t complicated: capture schemas through the REST API, capture configuration and references, restore in dependency order, and preserve or remap the IDs your messages depend on. The hard part is doing all of that together, in one artefact tied to your Kafka data backup, with the operational properties of real infrastructure.

Key takeaway: back up schema IDs, not just schema definitions, and prove your restore with a drill, because every Kafka message on disk points at an ID that must still resolve after recovery.

The teams who end up deleting unprocessed data weren’t careless. Schema Registry presented itself as boring, reliable infrastructure, and the wire format quietly made every Kafka message depend on a system they weren’t backing up properly. Whether you build your own tooling or use an existing tool, the test is the same: run the drill, verify the IDs and make sure messages still read on the other side.

Related reading:

• Building bulletproof disaster recovery for Apache Kafka
• Best managed Kafka services

Frequently asked questions