Apache Kafka outbox pattern

In the world of Apache Kafka, a seemingly innocent pattern is quietly corrupting data integrity across countless microservice architectures. It’s called “dual writes”, and it might be happening in your system right now.

As organisations decompose monolithic applications into microservices, each service typically maintains its own database. These services don’t exist in isolation: they need to collaborate and share information to solve business problems. This creates a critical challenge: how does a service update its own database while reliably communicating these changes to other services, often through message brokers like Apache Kafka?

Many developers intuitively reach for what seems like the simplest solution: update the database and then send a message to Kafka. This approach, known as “dual writes”, is fundamentally flawed and introduces significant risks to system consistency that can be costly and difficult to resolve.

Without atomic consistency between database updates and messaging, systems can fall into inconsistent states where data in one service doesn’t match related data in another. The consequences range from minor annoyances to critical business failures: orders that never ship, payments that aren’t recorded, or inventory that doesn’t reconcile.

This article demonstrates why dual writes are fundamentally flawed, explains the risks they introduce, and presents proven alternatives, above all the outbox pattern, for maintaining data consistency across distributed systems.

Updated October 2026: the Debezium connector example now uses current configuration names (topic.prefix, table.field.event.payload), and the KIP-939 section reflects its status as of October 2026.

What is the dual write problem?

What are dual writes?

Dual writes occur when an application needs to update two separate systems without transactional guarantees between them. In microservice architectures, this typically looks like:

1. Updating a database record (for example, creating a new order in the Orders service database)
2. Publishing a message to a messaging system like Kafka (for example, notifying the Shipment service about the new order)

These operations happen sequentially but without a distributed transaction spanning both systems. The pattern seems attractive because of its apparent simplicity, and the code looks clean and straightforward:

// Pseudo-code for dual write approach
try {
    // Step 1: Update database
    database.save(newOrder);

    // Step 2: Publish message
    kafka.publish("orders-topic", orderCreatedEvent);

    return success();
} catch (Exception e) {
    return error();
}

The fundamental flaw

The critical problem with dual writes is the lack of distributed transaction support between databases and messaging systems. Apache Kafka transactions cover writes to Kafka topics and consumer offsets, but Kafka cannot yet take part in a distributed transaction coordinated by a database. KIP-939 would change this, and we discuss it later.

This limitation creates two primary failure modes:

1. Database commits but messaging fails: your order is saved in the database, but the shipment service never learns about it. The customer’s order exists but will never be fulfilled.
2. Message is sent but database update fails: the shipment service receives notification about an order that doesn’t exist in the orders database. This could trigger processing of a non-existent order.

Both scenarios result in inconsistent system state. Even with retry mechanisms, there are edge cases that can’t be fully addressed, such as the service crashing between the two operations.

Why is consistency hard in distributed systems?

The CAP theorem context

The dual write problem is connected to the CAP theorem, which states that when a network partition occurs, a distributed system has to choose between consistency and availability. Because partitions can’t be ruled out, most modern distributed systems favour availability and settle for eventual consistency. We therefore need mechanisms that guarantee eventual consistency without data loss or corruption.

The cost of inconsistency can be substantial:

• Financial losses due to missed orders or duplicate processing
• Eroded customer trust when systems deliver unpredictable results
• Engineering time spent investigating and manually reconciling data
• Increased complexity in applications to handle potentially inconsistent states

Common failure scenarios

Several failure patterns repeatedly emerge in systems that rely on dual writes:

1. Transaction rollbacks creating orphaned messages: if a message is sent before the database transaction is committed, and that transaction then rolls back, the message refers to a state that never existed in the database.
2. Network partitions causing partial updates: network issues might allow the database update to succeed but prevent the message from being sent, creating an inconsistent state that’s difficult to detect and resolve.
3. Service outages creating backlogs: if the messaging system is temporarily unavailable, updates might continue in the database while messages queue up or fail entirely, leading to temporary or permanent inconsistency.
4. Ordering problems: without careful management, the order of operations might not be preserved, potentially violating business logic that depends on the sequence of events.

These failures don’t just create one-time issues. They compound over time, creating increasingly divergent system states that become harder to reconcile the longer they persist.

How do you detect and test for consistency issues?

Consistency issues can lurk beneath the surface, manifesting in subtle ways before causing major problems. Watch for these warning signs:

• Unexplained data discrepancies between services that should be in sync
• Customer complaints about missing actions or duplicate processing
• Reconciliation processes that consistently find exceptions requiring manual intervention
• Intermittent issues that can’t be reproduced reliably in testing environments
• “Ghost” records that appear in one system but not in related systems

If your team regularly performs “data fixing” operations or has built comprehensive reconciliation processes, you might be treating the symptoms rather than addressing the root cause.

Chaos engineering approaches

Proactively testing for consistency issues can help identify problems before they affect users:

1. Simulate network partitions: use tools like Toxiproxy or Chaos Monkey to introduce network failures between your application and your messaging system or database.
2. Test recovery mechanisms: force failures at various points in your transaction flow and verify that your system recovers to a consistent state.
3. Measure consistency under stress: introduce high load while degrading infrastructure performance, then measure how consistency holds up.

These tests often reveal that dual write approaches fail to maintain consistency under realistic failure conditions, highlighting the need for a more robust pattern.

What is the transactional outbox pattern?

The outbox pattern replaces the second write with a row in the service’s own database. In the same local transaction that changes the business data, the service inserts an event into an outbox table. Because both writes commit or roll back together, an event exists if and only if the business change was committed. A separate relay process then reads the outbox and publishes each event to Kafka.

A typical outbox table has these columns, which match the defaults of Debezium’s Outbox Event Router:

The relay delivers each event at least once, so consumers must be idempotent. The next section compares the two ways to build the relay.

How do you implement the outbox pattern with Kafka?

You need a reliable mechanism to move messages from your outbox table to Kafka. Let’s explore the two primary approaches through practical examples.

Polling-based implementation

The polling approach resembles how many teams initially implement the outbox pattern. Imagine an e-commerce company that has just implemented an outbox table for its order service:

// OrderService.java
@Transactional
public void createOrder(Order order) {
    // Save the order to the orders table
    orderRepository.save(order);

    // Create an outbox message
    OutboxEvent event = new OutboxEvent(
        UUID.randomUUID(),
        "Order",
        order.getId().toString(),
        objectMapper.writeValueAsString(orderToEventMapper.toEvent(order))
    );

    // Save the outbox message in the same transaction
    outboxRepository.save(event);
}

This works initially, but as order volume grows, the team encounters problems. During a flash sale, a long-running transaction creates an order with position 157, while several quick transactions create orders with positions 158, 159 and 160. The polling job picks up positions 158 to 160 first but misses 157 because it’s not yet committed. By the next polling cycle, it’s looking for positions above 160, completely skipping order 157.

The team also struggles with the polling frequency. When set to 5 seconds, customers complain about shipping notification delays. When reduced to 1 second, the database starts showing increased load during peak hours.

Log-based change data capture (CDC)

After encountering these issues, the team switches to log-based CDC using Debezium. They enable logical replication on Postgres and set up the Debezium PostgreSQL connector with the Outbox Event Router on Kafka Connect:

{
  "name": "orders-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "plugin.name": "pgoutput",
    "database.hostname": "postgres",
    "database.port": "5432",
    "database.user": "debezium",
    "database.password": "${file:/secrets/postgres.properties:password}",
    "database.dbname": "orders",
    "topic.prefix": "ecommerce",
    "table.include.list": "public.outbox_events",
    "transforms": "outbox",
    "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
    "transforms.outbox.table.field.event.key": "aggregate_id",
    "transforms.outbox.table.field.event.payload": "payload",
    "transforms.outbox.route.by.field": "aggregate_type",
    "transforms.outbox.route.topic.replacement": "${routedByValue}-events",
    "transforms.outbox.table.fields.additional.placement": "aggregate_id:header:aggregate_id"
  }
}

Since Debezium 2.0, topic.prefix replaces the old database.server.name setting. The table.field.* and route.by.field options are only needed because this outbox table uses aggregate_id and aggregate_type instead of Debezium’s default column names. Keep the database password in a Kafka Connect config provider rather than in the connector JSON.

With this configuration, Debezium reads changes directly from the database’s transaction log. When the team compares the old and new approaches during another flash sale, they see clear improvements:

• Events are captured in the order they were committed, so if Order A was committed before Order B, the events arrive in Kafka in that same order
• No events are missed, even during high-volume periods, because the transaction log contains every committed change
• Latency drops from seconds to milliseconds, as events are captured almost immediately after commit
• Database load decreases because there’s no need for frequent, potentially expensive queries

The team also simplifies its code. The order service looks almost identical, but the polling service is gone. Debezium handles extracting events and publishing them to Kafka.

Taking it a step further, Postgres users can remove the outbox table entirely, using pg_logical_emit_message() to write the event directly to the transaction log:

@Transactional
public void createOrder(Order order) {
    // Save the order to the orders table
    orderRepository.save(order);

    // Emit a transactional message directly to the write-ahead log
    String payload = objectMapper.writeValueAsString(orderToEventMapper.toEvent(order));
    jdbcTemplate.queryForObject(
        "SELECT pg_logical_emit_message(true, 'outbox', ?)",
        Object.class,
        payload
    );
}

Passing the payload as a bind parameter avoids building SQL by string concatenation. On the Debezium side, add the DecodeLogicalDecodingMessageContent transformation to decode the message content, then route it with the Outbox Event Router. This approach removes the need for outbox table housekeeping entirely.

What should you consider for production outbox implementations?

Housekeeping

Because the outbox table only serves as a temporary store for messages, you need a strategy to prevent it from growing indefinitely:

1. Transactional delete: insert and delete the outbox row in the same transaction. CDC still captures the insert from the log, and the Outbox Event Router automatically filters out DELETE operations on the outbox table.
2. Periodic cleanup: run a scheduled job to remove processed messages.
3. For Postgres users: use pg_logical_emit_message() to write directly to the transaction log without using a table at all.

Backfilling events

Sometimes you need to recreate events for existing data, for example after a misconfiguration or data loss. The outbox pattern can be extended to support this:

1. Use a “chunk” approach with marker events to delineate windows of backfilled data
2. Implement deduplication logic to discard backfill events for records that have received regular updates
3. Process these chunks efficiently while maintaining consistency with ongoing operations

Ensuring idempotency in consumers

Because the relay delivers events at least once, consumers need to handle potential duplicates:

1. Use a monotonically increasing value, such as the Postgres LSN (log sequence number), instead of arbitrary IDs
2. Consumers can track the highest processed LSN for each key or partition and ignore any messages with lower values
3. This approach requires storing only a single value rather than an ever-growing set of processed IDs

What are the alternatives to the outbox pattern?

While the outbox pattern is often the best solution, alternatives exist.

Event sourcing approach

Instead of writing to a database first, you could:

1. Write events directly to Kafka
2. Build a local read model by consuming these events

This works but has significant drawbacks:

• Loss of synchronous read-your-writes consistency
• Complexity in implementing constraints and validations
• More complex user experience, because events may not be immediately queryable

Two-phase commit with Kafka (KIP-939)

KIP-939 has been accepted and would let a Kafka producer take part in a two-phase commit with an external transaction coordinator, such as a database, which could make a direct dual-write approach safe. As of October 2026 it is not yet usable in an Apache Kafka release: its public APIs were removed from the 4.1 and 4.2 releases until the work is complete. Even once it ships:

• It increases availability requirements, because both the database and Kafka must be available for every write
• It adds overhead to your synchronous request processing
• It may not offer significant advantages over the outbox pattern for most use cases

Stream processing for legacy applications

For legacy applications that can’t be modified:

1. Use CDC to capture raw data changes
2. Apply stream processing, for example with Apache Flink, to transform these into higher-level events
3. Publish the transformed events to Kafka topics for consumption

This works well for systems you can’t change but makes transactional consistency more challenging to maintain.

What is the CDC outbox checklist?

1. Design your outbox table structure: include a message ID, aggregate type, aggregate ID and payload, and consider metadata for routing and tracking.
2. Select your relay approach: for most systems, log-based CDC with Debezium and its Outbox Event Router is recommended over polling.
3. Implement housekeeping: delete outbox rows in the same transaction or run a cleanup process. On Postgres, consider logical decoding messages instead of a table.
4. Ensure consumer idempotency: use LSNs or similar monotonically increasing values, or track processed event IDs.
5. Test thoroughly: verify behaviour under network partitions, confirm recovery after process crashes and validate ordering guarantees.

Key takeaway: never update the database and publish to Kafka as two separate writes. Write the event to an outbox in the same database transaction, relay it with log-based CDC such as Debezium’s Outbox Event Router, and make every consumer idempotent.

Frequently asked questions