The transactional outbox pattern with Debezium and Kafka

Every event-driven service eventually needs to do two things atomically: change its database and publish an event. Kafka transactions can’t span your database, and “save, then send” loses events when the process dies in between. The transactional outbox pattern, with Debezium reading the database log, is the standard, production-proven answer. This guide builds it end to end with Spring Boot, PostgreSQL, Debezium and Kafka.

The dual-write problem

@Transactional
public void placeOrder(Order order) {
    orders.save(order);
    kafka.send("orders", order.getId(), new OrderPlaced(order));   // not part of the DB transaction
}

This code has three failure modes, and all of them happen in production eventually:

• Send succeeds, commit fails — consumers see an order that doesn’t exist.
• Commit succeeds, send fails (broker down, pod killed) — the order exists but nobody is ever told.
• Ordering breaks — two transactions can commit in one order and publish in the other.

Moving the send after the commit (for example with @TransactionalEventListener) only narrows the window; a crash between commit and send still loses the event.

How the outbox pattern works

1. In the same database transaction as the business change, insert a row into an outbox table describing the event.
2. A separate relay reads committed outbox rows and publishes them to Kafka. With change data capture (CDC), the relay is Debezium tailing the database’s write-ahead log.
3. Consumers process events idempotently, because delivery is at-least-once.

The database commit becomes the single source of truth: if the order row is committed, so is its event, and Debezium will publish it — even if your service crashes a millisecond later.

The outbox table and write path

Use the column names Debezium’s outbox router expects by default:

CREATE TABLE outbox (
    id            uuid         PRIMARY KEY,
    aggregatetype varchar(255) NOT NULL,   -- routes to a topic, e.g. "order"
    aggregateid   varchar(255) NOT NULL,   -- becomes the Kafka message key
    type          varchar(255) NOT NULL,   -- event type, e.g. "OrderPlaced"
    payload       jsonb        NOT NULL,
    created_at    timestamptz  NOT NULL DEFAULT now()
);

@Service
class OrderService {
    private final OrderRepository orders;
    private final OutboxRepository outbox;
    private final ObjectMapper json;

    // constructor omitted

    @Transactional
    public void placeOrder(Order order) {
        orders.save(order);
        outbox.save(new OutboxEvent(
            UUID.randomUUID(),
            "order",
            order.getId(),
            "OrderPlaced",
            json.valueToTree(new OrderPlaced(order.getId(), order.getTotal()))
        ));
        // no Kafka call here: one local transaction, all or nothing
    }
}

Because the message key is aggregateid, all events for one order land on the same partition and stay in commit order.

Configuring Debezium’s outbox router

PostgreSQL needs logical replication enabled (wal_level=logical; on Amazon RDS, set rds.logical_replication=1 in the parameter group). Then register the connector with Kafka Connect:

{
  "name": "orders-outbox",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "plugin.name": "pgoutput",
    "database.hostname": "orders-db",
    "database.port": "5432",
    "database.user": "debezium",
    "database.password": "${file:/secrets/db.properties:password}",
    "database.dbname": "orders",
    "topic.prefix": "orders",
    "table.include.list": "public.outbox",
    "slot.name": "orders_outbox",
    "publication.autocreate.mode": "filtered",

    "transforms": "outbox",
    "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
    "transforms.outbox.route.topic.replacement": "${routedByValue}.events",
    "transforms.outbox.table.expand.json.payload": "true",
    "transforms.outbox.table.fields.additional.placement": "type:header:eventType",

    "heartbeat.interval.ms": "10000",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false"
  }
}

What the Event Router does with each committed outbox insert:

• Routes it to a topic built from aggregatetype — here order.events.
• Uses aggregateid as the message key and payload as the message value — the clean event, not Debezium’s change-event envelope.
• Puts the event id in a message header, plus the event type as the eventType header from the placement setting above.

Debezium reads the write-ahead log, not the table, so rows can be deleted as soon as they are committed. A scheduled cleanup (DELETE FROM outbox WHERE created_at < now() - interval '1 day') keeps the table small; some teams insert and delete in the same transaction, which still produces the event.

Idempotent consumers

Debezium guarantees at-least-once delivery: after a connector restart, a few events can be published again. Consumers must deduplicate, and the event id header is the natural key. Record processed ids in the same transaction as the consumer’s own changes:

@KafkaListener(topics = "order.events", groupId = "billing")
@Transactional
public void onEvent(@Payload OrderPlaced event,
                    @Header("id") String eventId,
                    @Header("eventType") String type) {
    // INSERT ... ON CONFLICT DO NOTHING returns 0 rows when we've seen this id before
    if (processed.markIfNew(eventId) == 0) {
        return;   // duplicate: already handled
    }
    invoices.createFor(event);
}

CREATE TABLE processed_event (
    event_id     varchar(64) PRIMARY KEY,
    processed_at timestamptz NOT NULL DEFAULT now()
);

Running it in production

• Watch the replication slot. If the connector stops, PostgreSQL keeps all WAL since the slot’s position, and the disk fills. Alert on slot lag:

SELECT slot_name,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS lag
FROM pg_replication_slots;

• Heartbeats matter on quiet databases. If the outbox table is idle while other tables are busy, the slot can’t advance; heartbeat.interval.ms lets Debezium confirm progress regularly.
• Drop the slot when you remove a connector (SELECT pg_drop_replication_slot('orders_outbox')), or it retains WAL forever.
• Failover: logical replication slots must survive a primary failover. Check that your PostgreSQL version and platform sync slots to the standby, or plan for a connector re-snapshot.
• Version the payload. The event is now a public contract; add fields compatibly and use the type header to evolve deliberately.

Alternatives to CDC

If you already run Kafka Connect, Debezium is the cleanest option. If you don’t, a polling publisher with SKIP LOCKED gives the same guarantee with only your application and database — the outbox table and idempotent consumers stay exactly the same, so you can switch to CDC later.