The Outbox Pattern

The claim under test

This is the most common shape in enterprise event-driven code. It appears in every tutorial, it passes every code review, and it is wrong in a way that unit tests cannot reach.

@Transactional
public Order place(PlaceOrder cmd) {
    Order order = repository.save(Order.from(cmd));      // system 1: the database
    publisher.publish(new OrderPlaced(order.getId()));   // system 2: the broker
    return order;
}

Line three is a local transaction. Line four is a network call to a completely different piece of infrastructure, with its own availability, its own failure modes and its own idea of what has happened. @Transactional has authority over the first and none over the second: the broker is not enrolled in the transaction, cannot be told to roll back, and will not undo a message it has already accepted.

So line four is a distributed transaction in disguise. The method claims an atomic outcome, the order exists and the world was told, across two resources that have no shared commit protocol. Chris Richardson states the consequence without hedging, and the important part is that it holds in both directions:

Two sentences, two orderings, two different bugs. Reordering the statements does not remove the problem; it only chooses which failure you get.

The dual-write windows

Draw it as a timeline and the argument stops being abstract. There are exactly two orderings available to you, each has an interval between the two writes, and a process can die inside that interval.

The two outcomes are worth naming separately, because they are found by different people at different times and they cost different amounts to repair.

Why not two-phase commit

There is a protocol designed for exactly this, and dismissing it in one line is the standard mistake. XA and two-phase commit are real, mature, and still in production service: a transaction manager asks every participant to prepare, and only when all of them have durably promised does it ask them all to commit. In a JTA container with an XA-capable relational database and an XA-capable JMS provider, the code in the first section of this page is genuinely correct. That configuration is not a myth; it ran the enterprise for two decades.

Richardson still rules it out first, and states the reason as a force rather than an opinion:

The first clause is decisive in practice, and the Debezium team put the specific case plainly when introducing the pattern:

The honest summary is not "2PC is bad". It is that 2PC solves the problem only when every participant speaks it, and in a modern estate the message broker usually does not. The outbox reaches the same guarantee by removing the second participant entirely.

The outbox, structurally

The move is almost embarrassingly simple once you see it: stop trying to write to two systems and write to one. The message becomes a row, the row goes into the same transaction as the business change, and something else takes it from there.

The benefit Richardson claims for this is precise, and it is the sentence to quote in a design review: messages "are guaranteed to be sent if and only if the database transaction commits". Both halves matter. No commit, no message, so the phantom is gone. Commit, then eventually a message, so the loss is gone.

One discipline goes with the table. The outbox is a queue that happens to be stored in your database, and treating it as anything else breaks the log-based relay:

Richardson also lists the pattern's one real drawback, and it is a human one: "Potentially error prone since the developer might forget to publish the message/event after updating the database." A service method that writes a business row and no outbox row compiles, passes its tests, and silently drops the event forever. This is worth an architecture test or a code-review rule, not a wiki page.

Two ways to move the message

The outbox splits the problem in half; the second half is getting rows out of the table and into the broker. There are two named patterns for it and the choice is genuinely a trade, not a maturity ladder.

Polling Publisher

A scheduled loop selects undelivered rows, publishes them, and marks them delivered. That is the entire mechanism. It needs no new infrastructure, no connector, no additional cluster, and it works on any SQL database you already run. It costs you a floor on latency, because no event can be delivered faster than the poll interval, and a query that runs forever whether or not there is anything to do. Multiple relay instances need SKIP LOCKED to avoid publishing the same row twice.

Transaction Log Tailing (CDC)

Instead of asking the database what changed, you read the record it already keeps of what changed: the PostgreSQL write-ahead log, the MySQL binlog, a DynamoDB table stream. Debezium is the usual implementation, and its Outbox Event Router transformation reshapes each captured outbox row into a message routed by aggregatetype and keyed by aggregateid. Latency drops to the propagation delay of the log, and the polling load disappears entirely. In exchange you get a connector, a Kafka Connect cluster or equivalent to run it in, replication-slot configuration, and a component whose lag is now a thing you must monitor and whose failure mode is database-specific.

What the outbox does not give you

It gives you delivery. It does not give you exactly once, and any design that assumes otherwise is broken in a way that will only appear under failure. The pattern's own Issues section states the reason:

Richardson's conclusion from that is unambiguous: the consumer must be idempotent, tracking the IDs of the messages it has already processed. This is the same window as the one at the top of this page, publish then record, moved one component to the right, and it is irreducible for the same reason. The difference is that here the failure is a duplicate rather than a loss, and a duplicate is something a consumer can be built to absorb.

That makes the outbox and the idempotent consumer a matched pair, not two independent choices. Everything in Message Brokers vs Event Buses about at-least-once delivery applies here unchanged: the id must be stamped by the producer rather than derived from broker metadata, the deduplication claim and the business write must share one transaction, and the dedupe table needs a retention policy longer than the maximum redelivery delay. The outbox is what makes at-least-once reachable from a database write; that page is what makes it safe once it arrives.

The Inbox pattern

The receiving-side mirror image is sometimes called the inbox: before doing any work, the consumer inserts the message id into an inbox table with a unique constraint, in the same transaction as the work itself. A duplicate delivery loses the race on the constraint and returns without acting. The outbox makes a write reliably publishable; the inbox makes a delivery reliably applied once. Neither is sufficient alone, and the pair is what people mean when they say effectively-once. Debezium's router leans on the same identifier for this: the outbox id column "Contains the unique ID of the event. In an outbox message, this value is a header."

Ordering, honestly

Ordering is where outbox write-ups tend to overpromise. Richardson lists it as a force rather than a benefit, and note how carefully he scopes it:

The same aggregate, not the whole system. That is the guarantee to aim for, and three things have to line up to get it. A monotonic sequence_no in the outbox gives the relay a total order to read in. Concurrent writers to one aggregate serialise on that aggregate's row lock, so their outbox rows are assigned in the order their transactions actually committed. And the relay must publish with the aggregate id as the partition key, or the broker will re-order what the relay carefully ordered.

The gap this would close

Doing it properly

Four tabs, in the order the change actually happens. The first is what is being deleted; the second is the table that replaces it; the third is the write path, which is where the whole guarantee lives; the fourth is the relay and the consumer that has to tolerate its retries. Everything is PostgreSQL, because SKIP LOCKED syntax is dialect-specific, and it exists in PostgreSQL 9.5+, MySQL 8.0+ and Oracle, and does not exist in SQL Server, where the equivalent is the READPAST table hint.

@Slf4j
@Service
@RequiredArgsConstructor
public class OrderService {

    private final OrderRepository orders;
    private final DomainEventPublisher publisher;
    private final TransactionTemplate txTemplate;

    /**
     * Ordering 1: publish inside the transaction. The message leaves the
     * process before the COMMIT that would justify it, so a constraint
     * violation, a deadlock or a connection reset on the way out produces a
     * phantom: consumers have reacted to an order that does not exist.
     */
    @Transactional
    public Order placeThenCommit(PlaceOrder cmd) {
        Order order = orders.save(Order.from(cmd));
        publisher.publish(new OrderPlaced(order.getId(), order.total()));
        return order;                     // the COMMIT happens after this line
    }

    /**
     * Ordering 2: publish after the transaction commits. Now the row is
     * durable and the message is not. A try/catch around the publish does not
     * help: the failure this is about is the JVM being killed between the two
     * statements, and no handler runs in a process that no longer exists.
     */
    public Order commitThenPublish(PlaceOrder cmd) {
        Order order = txTemplate.execute(tx -> orders.save(Order.from(cmd)));
        try {
            publisher.publish(new OrderPlaced(order.getId(), order.total()));
        } catch (BrokerException e) {
            // Retrying here narrows the window. It does not close it, and a
            // retry loop that outlives the request thread is a second bug.
            log.error("event lost for order {}", order.getId(), e);
        }
        return order;
    }
}

-- PostgreSQL. The outbox table lives in the SAME database as the business
-- tables. That is the whole mechanism: one connection, one transaction, one
-- COMMIT, and therefore no second system that can fail independently.
CREATE TABLE outbox (
    sequence_no       BIGINT       GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    message_id        UUID         NOT NULL UNIQUE,
    aggregate_type    VARCHAR(255) NOT NULL,
    aggregate_id      VARCHAR(255) NOT NULL,
    type              VARCHAR(255) NOT NULL,
    payload           JSONB        NOT NULL,
    created_at        TIMESTAMPTZ  NOT NULL DEFAULT now(),
    delivered_at      TIMESTAMPTZ,
    dead_lettered_at  TIMESTAMPTZ,
    attempts          INT          NOT NULL DEFAULT 0
);

-- The relay asks exactly one question: what is still undelivered? A partial
-- index keeps that query proportional to the backlog rather than to the
-- table, which matters because the table only ever grows.
CREATE INDEX outbox_pending_idx
    ON outbox (sequence_no)
 WHERE delivered_at IS NULL
   AND dead_lettered_at IS NULL;

-- message_id is UNIQUE because it is the consumer's deduplication key. It is
-- stamped by the producer, once, and never re-derived from broker metadata:
-- a broker-assigned id changes on redelivery and silently defeats dedupe.

@Entity
@Table(name = "outbox")
public class OutboxMessage {

    /** Insertion order, assigned by the database. Not a global clock. */
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    @Column(name = "sequence_no")
    private Long sequenceNo;

    /** The consumer's deduplication key. Stamped by the producer, once. */
    @Column(name = "message_id", nullable = false, unique = true)
    private UUID messageId;

    @Column(name = "aggregate_type", nullable = false)
    private String aggregateType;

    @Column(name = "aggregate_id", nullable = false)
    private String aggregateId;

    @Column(nullable = false)
    private String type;

    /** Hibernate 6 maps a String to jsonb with an explicit JDBC type code. */
    @JdbcTypeCode(SqlTypes.JSON)
    @Column(nullable = false, columnDefinition = "jsonb")
    private String payload;

    @Column(name = "created_at", nullable = false)
    private Instant createdAt;

    @Column(name = "delivered_at")
    private Instant deliveredAt;

    @Column(name = "dead_lettered_at")
    private Instant deadLetteredAt;

    @Column(nullable = false)
    private int attempts;

    protected OutboxMessage() { }   // required by JPA

    public static OutboxMessage of(String aggregateType, String aggregateId,
                                   String type, UUID messageId, String payload) {
        OutboxMessage m = new OutboxMessage();
        m.messageId = messageId;
        m.aggregateType = aggregateType;
        m.aggregateId = aggregateId;
        m.type = type;
        m.payload = payload;
        m.createdAt = Instant.now();
        return m;
    }

    void markDelivered(Instant at) { this.deliveredAt = at; }
    void recordAttempt()           { this.attempts++; }
    void deadLetter(Instant at)    { this.deadLetteredAt = at; }
}

@Service
@RequiredArgsConstructor
public class OrderService {

    private final OrderRepository orders;
    private final OutboxRepository outbox;
    private final ObjectMapper json;

    /**
     * One transaction. Two INSERTs. One COMMIT. There is no publisher in this
     * class and no broker in this transaction. The only durable side effect
     * is a pair of rows in one database, written together or not at all.
     */
    @Transactional
    public Order place(PlaceOrder cmd) {
        Order order = orders.save(Order.from(cmd));

        OrderPlaced event = new OrderPlaced(
                UUID.randomUUID(), order.getId(), order.total(), Instant.now());

        outbox.save(OutboxMessage.of(
                "Order",             // aggregate type, becomes the topic
                order.getId(),       // aggregate id, becomes the partition key
                "OrderPlaced",
                event.eventId(),
                toJson(event)));

        return order;
    }

    /**
     * Deliberately unchecked, and this is not a style preference. Spring's
     * default rollback rule covers RuntimeException and Error only: a checked
     * exception escaping a transactional method COMMITS. Declaring
     * JsonProcessingException on place(...) would persist the order, lose the
     * outbox row, and reinstate the dual write through a framework default.
     */
    private String toJson(Object event) {
        try {
            return json.writeValueAsString(event);
        } catch (JsonProcessingException e) {
            throw new UncheckedIOException(e);
        }
    }
}

public interface OutboxRepository extends JpaRepository<OutboxMessage, Long> {

    /**
     * PostgreSQL. FOR UPDATE takes a row lock on everything the SELECT returns;
     * SKIP LOCKED tells the database to step over rows another relay instance
     * already holds instead of queueing behind them. Without FOR UPDATE two
     * instances publish the same message; without SKIP LOCKED the second
     * instance simply blocks, and you have one relay wearing two pods.
     *
     * The predicate is a flag, not a cursor. sequence_no is assigned at INSERT
     * and becomes visible at COMMIT, so a lower number can appear after a
     * higher one has already been read. A high-water mark would skip that row
     * forever; delivered_at IS NULL cannot.
     */
    @Query(nativeQuery = true, value = """
            SELECT * FROM outbox
             WHERE delivered_at IS NULL
               AND dead_lettered_at IS NULL
             ORDER BY sequence_no
             LIMIT :batch
             FOR UPDATE SKIP LOCKED
            """)
    List<OutboxMessage> lockPending(@Param("batch") int batch);
}

@Slf4j
@Component
@RequiredArgsConstructor
public class OutboxRelay {

    private static final int BATCH = 100;
    private static final int MAX_ATTEMPTS = 10;

    private final OutboxRepository outbox;
    private final MessageBroker broker;

    /**
     * Publish, then mark delivered. A crash between those two statements
     * republishes on the next tick, which is the at-least-once the pattern
     * promises, and the reason the consumer below must deduplicate. Marking
     * first would make the relay at-most-once and lose the event one layer down.
     *
     * The row locks are held for the life of this transaction, so the method
     * must be transactional and must stay short. A slow broker here is a long
     * lock, and a long lock is a stalled outbox.
     */
    @Scheduled(fixedDelayString = "${outbox.poll-interval-ms:500}")
    @Transactional
    public void drain() {
        for (OutboxMessage m : outbox.lockPending(BATCH)) {
            try {
                broker.publish(
                        m.getAggregateType(),        // topic
                        m.getAggregateId(),          // partition key, per-aggregate order
                        Map.of("id", m.getMessageId().toString()),
                        m.getPayload());
                m.markDelivered(Instant.now());
            } catch (BrokerException e) {
                m.recordAttempt();
                if (m.getAttempts() >= MAX_ATTEMPTS) {
                    m.deadLetter(Instant.now());     // park it; stop blocking the queue
                    log.error("outbox message {} parked after {} attempts",
                            m.getMessageId(), MAX_ATTEMPTS, e);
                } else {
                    break;                           // do not publish past a gap
                }
            }
        }
    }
}

@Component
@RequiredArgsConstructor
public class OrderPlacedConsumer {

    private static final String CONSUMER = "shipping-projection";

    private final InboxRepository inbox;
    private final ShipmentPlanner planner;

    /**
     * The outbox guarantees the message arrives. It never guarantees it
     * arrives once. The deduplication claim and the business write share one
     * transaction: a "seen" flag written after the work is a smaller race
     * window, not idempotence.
     */
    @Transactional
    @KafkaListener(topics = "Order.events", groupId = CONSUMER)
    public void on(@Header("id") UUID messageId, OrderPlaced event) {
        if (inbox.claim(messageId, CONSUMER) == 0) {
            return;                     // an earlier delivery already applied this
        }
        planner.planFor(event.orderId());
    }
}

/** The Inbox pattern: the outbox's mirror image on the receiving side. */
public interface InboxRepository extends JpaRepository<InboxMessage, UUID> {

    /**
     * Returns 1 when this delivery claimed the message, 0 when a previous one
     * did. ON CONFLICT DO NOTHING rather than catching the constraint
     * violation: a violation raised inside an active persistence context marks
     * the transaction rollback-only, so catching it and carrying on does not
     * work. The composite key lets each consumer deduplicate independently.
     */
    @Modifying
    @Query(nativeQuery = true, value = """
            INSERT INTO inbox (message_id, consumer, received_at)
            VALUES (:messageId, :consumer, now())
            ON CONFLICT (message_id, consumer) DO NOTHING
            """)
    int claim(@Param("messageId") UUID messageId, @Param("consumer") String consumer);
}

Operational notes

The outbox is the rare pattern whose tutorials are complete about the happy path and silent about the parts that page you. Three of them matter more than the rest.

The table grows forever

Every message your service has ever emitted is a row, and marking a row delivered does not make it smaller. At a thousand events a minute that is half a billion rows a year, an index that no longer fits in cache, autovacuum falling behind, and a poll query that gets slower every week. Cleanup is not housekeeping; it is part of the design. Delete or archive delivered rows on a schedule, or partition by day and drop whole partitions, and dropping a partition is a metadata operation, whereas a large DELETE generates the dead tuples that create the next problem.

With a log-based relay you can go further and delete each row in the same transaction that inserted it: the connector reads the log, not the table, so the INSERT is captured even though the row never survives to be queried. Debezium's write-up of the pattern notes exactly this: "The calls to persist() and remove() will create an INSERT and a DELETE entry in the log once the transaction commits." The table stays permanently empty and the messages still flow. This trick is unavailable to a polling relay, which needs the row to still be there to find it.

Relay lag is the metric, not relay liveness

A relay process that is running, healthy, and publishing nothing looks identical to a healthy idle system on every dashboard that measures uptime. The signal that distinguishes them is the age of the oldest undelivered row: it is near zero when things are fine and grows without bound the moment they are not. Alert on that, and on the count of dead-lettered rows, which should be zero and is otherwise a queue of things nobody has looked at.

Poison messages block the queue

A message that cannot be published, whether a payload the serializer rejects, a topic that does not exist, or a record above the broker's size limit, will be retried on every tick forever. If the relay preserves ordering by refusing to publish past a failure, that single row now blocks every message behind it, and the outage is total rather than partial. Cap the attempts, park the row, alert, and let the queue drain. Parking is a deliberate consistency decision, not a cleanup: you have chosen to skip a message to keep the rest moving, and somebody has to be told.

• Archive or partition delivered rows. A retention job from day one, not after the first slow query. Dropping a partition beats deleting a million rows.
• Alert on the age of the oldest undelivered row. Process liveness proves nothing; a stalled relay is a healthy process doing nothing.
• Cap attempts and park poison messages, then alert on the parked count. An unwatched dead-letter state is data loss with a nicer name.
• Decide the ordering trade explicitly. One relay with leader election, or shard by aggregate id, but write down which one you chose and why.
• Keep the relay transaction short. It holds row locks for its whole duration; a broker timeout inside it stalls every other relay instance.
• Give the inbox table a retention policy too, on a window comfortably longer than the maximum redelivery delay.
• Publishing from the service and writing to the outbox "just to be safe". That is the dual write again, now with duplicates as well.
• An outbox in a different database from the business tables. Two databases is two transactions; the entire guarantee is that there is only one.
• Reaching for CDC before you have measured that the poll interval is actually the problem.

Canonical sources

• Chris Richardson: Transactional Outbox · the problem statement, the 2PC force, the solution sentence, the ordering force, and the idempotency issue. Every Richardson quote on this page is from here.
• Chris Richardson: Polling Publisher · the relay variant that needs no new infrastructure.
• Chris Richardson: Transaction Log Tailing · the CDC variant, with the per-database implementations named (MySQL binlog, Postgres WAL, DynamoDB streams).
• Debezium: Outbox Event Router · the expected outbox columns, the id-as-header rule, and the insert-only discipline the table depends on.
• Debezium: Reliable Microservices Data Exchange With the Outbox Pattern · why Kafka cannot be enlisted in an XA transaction, and the insert-then-delete trick for keeping the table empty.
• Message Brokers vs Event Buses · the delivery-semantics groundwork this page assumes: at-most-once versus at-least-once, and why idempotent consumption is the price of the latter.