The outbox pattern closes that gap.

The Dual-Write Problem

Say the flight reservation step succeeds. You update your database to mark the reservation as confirmed and then publish a FlightReserved event to your message broker. These are two operations against two different systems. There’s no way to make them atomic without 2PC – which is exactly what sagas are trying to avoid.

If you write to the database first and then crash before publishing, the event is never delivered. The hotel reservation step never starts. The saga stalls. Depending on your monitoring, this might sit undetected for a while.

If you publish first and then crash before writing to the database, you’ve told the hotel step to proceed even though the flight reservation isn’t durably recorded. The flight service comes back up with no record of the reservation. You’ve introduced a phantom.

The Outbox

The fix: write the event to an outbox table in the same database, in the same local transaction as your state update. Then have a separate process relay events from the outbox to the message broker.

BEGIN TRANSACTION
  UPDATE flights SET status = 'reserved' WHERE id = ?
  INSERT INTO outbox (event_type, payload, created_at)
    VALUES ('FlightReserved', '{"booking_id": 123, ...}', NOW())
COMMIT

The state change and the event are now atomic with respect to the database. Either both are committed or neither is. The outbox row is durable.

A separate relay process – sometimes called the outbox processor or message relay – polls the outbox table, publishes each event to the message broker, and marks the row as processed once delivery is confirmed.

Relay process:
  SELECT * FROM outbox WHERE processed_at IS NULL ORDER BY created_at LIMIT 100
  FOR EACH row:
    publish to message broker
    UPDATE outbox SET processed_at = NOW() WHERE id = row.id

At-Least-Once Delivery

The relay delivers at-least-once. If the relay crashes after publishing but before marking the row as processed, it will publish the same event again on restart. Downstream consumers have to be idempotent – processing the same event twice should produce the same result as processing it once.

This is a real design constraint. Your hotel reservation step needs to check whether it already processed a given FlightReserved event before creating a new reservation. Deduplication is typically done with a unique constraint on the event ID or booking ID in the receiving service’s database.

At-least-once with idempotent consumers is the standard contract in event-driven systems. Exactly-once is theoretically possible with additional coordination but practically rare – most systems accept at-least-once and invest in idempotency instead.

Change Data Capture as an Alternative Relay

Polling the outbox table works but adds load to the database. An alternative is Change Data Capture (CDC): instead of a polling process, you read directly from the database replication log.

Tools like Debezium connect to PostgreSQL’s logical replication stream or MySQL’s binlog and emit a change event for every row insert. Point Debezium at the outbox table and it publishes events automatically, with no polling query needed.

CDC-based relay has lower overhead and lower latency. The tradeoff: it’s another piece of infrastructure to run and operate. For smaller deployments, polling is simpler. For high-throughput systems where polling lag or database load is a concern, CDC is worth it.

Saga State Persistence

Related to the outbox is the question of where saga state itself lives.

The orchestrator driving the saga needs to know the current step, which steps have completed, and what the compensating actions are if something fails. Two common approaches:

Dedicated saga store: the orchestrator writes saga state to its own database table after each step. Simple, queryable, easy to inspect when debugging a stuck saga. The outbox pattern applies here too – saga state update and event publish happen in the same transaction.

Event sourcing: saga state is derived from the event log itself. There’s no explicit state table – you reconstruct the saga’s current position by replaying events. More complex to implement but gives you a full audit trail of everything that happened and an easy path to replaying failed sagas.

Most teams start with a dedicated saga store and move to event sourcing only if they need the audit trail or replay capability.

What Fails in Compensation

One more gap worth naming: compensating transactions can fail too.

You cancel the hotel reservation as part of rolling back a failed booking. The cancel call times out. Now your saga is stuck in a partially compensated state – the flight is cancelled but the hotel reservation is still live.

The standard answer is retry with backoff. Compensating transactions need to be idempotent for the same reason forward steps do. The outbox pattern helps here too – compensation requests go through the same outbox mechanism, so they’re durably enqueued even if the downstream service is temporarily unavailable.

If compensation keeps failing after retries, the saga is in a “saga failure” state that requires human intervention or a dead-letter queue. Building an operator UI for stuck sagas is not glamorous work, but it’s necessary in any production saga implementation.

The outbox pattern doesn’t make sagas simple. It removes the dual-write problem and gives you reliable event delivery – which turns a fundamentally broken approach into one that’s merely complex.

The Saga pattern is the alternative. You give up true atomicity in exchange for availability, scalability, and no cross-service locks.

How It Works

Break one big distributed transaction into a sequence of local transactions, each owned by a single service. Each step publishes an event or sends a message when it completes. If any step fails, run compensating transactions in reverse to undo the completed steps.

The classic example is a trip booking:

Step 1: Reserve flight    → local tx in Flight Service
Step 2: Reserve hotel     → local tx in Hotel Service
Step 3: Charge card       → local tx in Payment Service

If step 3 fails:

FORWARD:   [Flight ✓] → [Hotel ✓] → [Payment ✗]
ROLLBACK:              ← [Cancel Hotel] ← [Cancel Flight]

No global rollback. No coordinator holding locks across all three services. Each service knows how to undo its own step.

Two Flavors

Choreography – each service listens for events and decides what to do next based on what it hears. No central coordinator. Fully decoupled. Easy to start, and a debugging nightmare once you have more than a handful of services. Tracing a failed saga means reconstructing a sequence of events scattered across multiple event streams.

Orchestration – a central saga orchestrator drives the sequence. It tells each service what to do, collects results, and issues compensating calls on failure. Full visibility into saga state. Easier to reason about and monitor. The orchestrator becomes a coordination hub – not quite a single point of failure if you make it durable, but a complexity center. Most teams end up here in production.

The Tradeoff You Can’t Avoid

Sagas trade the “I” in ACID (isolation) for availability. While a saga is in-flight, intermediate state is visible to other parts of the system. A hotel room might be reserved even though the flight hasn’t been confirmed yet. Other services can observe that state.

You have to design for this explicitly:

• Eventual consistency – the system converges to a correct final state, but not atomically
• Idempotent compensating transactions – if your “cancel flight” call gets retried twice due to a network hiccup, it must be safe to run twice. Compensation that isn’t idempotent will make your failure recovery worse than the original failure.

Where It Shows Up

Basically every large-scale order or booking flow:

• Uber – ride booking, driver assignment, payment charge, all as separate service transactions
• Amazon – order placement, inventory reservation, fulfillment, shipping
• Netflix – subscription management, content licensing flows

These systems can’t afford to hold cross-service locks for the duration of a user-facing transaction. They accept that intermediate states exist and design their compensating logic carefully.

The saga pattern doesn’t make distributed transactions easy. It makes the tradeoffs explicit and gives you a way to build systems that stay available even when individual steps fail.