The name comes from ship design. A ship’s hull is divided into watertight compartments. One compartment floods, the others stay dry. The ship stays afloat. Without bulkheads, one breach sinks everything.

The Problem Without Bulkheads

Imagine a service that calls three downstream APIs – payment, inventory, and recommendations. You have a thread pool with 100 threads handling all outbound requests.

The recommendations API starts responding slowly. Requests pile up waiting. Threads get tied up waiting for responses that take 10 seconds instead of 100ms. Within minutes all 100 threads are occupied waiting on recommendations. Now payment requests and inventory requests can’t get a thread either. Your service is down for all three functions because one non-critical API got slow.

A circuit breaker would help here once the recommendations API is actually failing – but slow isn’t failing. The requests are completing, just slowly. The circuit never trips. The thread pool still drains.

The Fix: Separate Resource Pools

Partition your resources so that one downstream dependency can only consume its own allocation.

Instead of one pool of 100 threads, you have three pools:

• Payment: 40 threads
• Inventory: 40 threads
• Recommendations: 20 threads

The recommendations API goes slow. It consumes all 20 of its threads. Payment and inventory are unaffected – they have their own pools. The non-critical dependency is isolated in its own compartment.

Two Implementation Styles

Thread pool isolation is what Hystrix made popular. Each downstream call goes through a dedicated thread pool. Callers block on their own pool, not a shared one. Overhead: context switches between the caller thread and the pool thread for every call.

Semaphore isolation is lighter. Instead of separate thread pools, you put a semaphore in front of each dependency with a max concurrent count. Callers execute on their own thread but must acquire the semaphore first. No thread switching overhead. The tradeoff: if the call actually blocks the thread (synchronous I/O), you’re still tying up the caller’s thread. Better for fast calls, worse for slow ones.

For services making lots of downstream calls with variable latency, thread pool isolation is generally safer. The overhead is real but the isolation is stronger.

Combining with Circuit Breakers

Bulkheads and circuit breakers are complementary, not alternatives. They protect against different failure modes:

• Bulkhead: limits resource consumption from a slow or partially degraded dependency
• Circuit breaker: stops calls entirely when a dependency crosses a failure threshold

The typical pattern is both together. The bulkhead caps how many threads/connections one dependency can consume. The circuit breaker trips when that dependency starts returning errors. Between the two, you handle the full range of failure behavior from “slightly slow” to “completely down.”

Netflix Hystrix combined both in a single abstraction. Resilience4j does the same. In a service mesh like Istio, connection pool settings are the bulkhead and outlier detection is the circuit breaker – same ideas, configured at the mesh level rather than the application level.

What to Size the Pools At

This is the hard part. Too small and you’re rejecting valid traffic. Too large and the isolation breaks down.

A reasonable starting point: look at your normal concurrency for each dependency, add headroom for spikes, and set the max at roughly 2x your normal peak. Monitor pool utilization in steady state – if the recommendations pool is regularly at 18/20, you either need to size up or your recommendations calls are too slow.

Bulkheads don’t fix slow dependencies. They contain the blast radius while you do.

The outbox pattern closes that gap.

The Dual-Write Problem

Say the flight reservation step succeeds. You update your database to mark the reservation as confirmed and then publish a FlightReserved event to your message broker. These are two operations against two different systems. There’s no way to make them atomic without 2PC – which is exactly what sagas are trying to avoid.

If you write to the database first and then crash before publishing, the event is never delivered. The hotel reservation step never starts. The saga stalls. Depending on your monitoring, this might sit undetected for a while.

If you publish first and then crash before writing to the database, you’ve told the hotel step to proceed even though the flight reservation isn’t durably recorded. The flight service comes back up with no record of the reservation. You’ve introduced a phantom.

The Outbox

The fix: write the event to an outbox table in the same database, in the same local transaction as your state update. Then have a separate process relay events from the outbox to the message broker.

BEGIN TRANSACTION
  UPDATE flights SET status = 'reserved' WHERE id = ?
  INSERT INTO outbox (event_type, payload, created_at)
    VALUES ('FlightReserved', '{"booking_id": 123, ...}', NOW())
COMMIT

The state change and the event are now atomic with respect to the database. Either both are committed or neither is. The outbox row is durable.

A separate relay process – sometimes called the outbox processor or message relay – polls the outbox table, publishes each event to the message broker, and marks the row as processed once delivery is confirmed.

Relay process:
  SELECT * FROM outbox WHERE processed_at IS NULL ORDER BY created_at LIMIT 100
  FOR EACH row:
    publish to message broker
    UPDATE outbox SET processed_at = NOW() WHERE id = row.id

At-Least-Once Delivery

The relay delivers at-least-once. If the relay crashes after publishing but before marking the row as processed, it will publish the same event again on restart. Downstream consumers have to be idempotent – processing the same event twice should produce the same result as processing it once.

This is a real design constraint. Your hotel reservation step needs to check whether it already processed a given FlightReserved event before creating a new reservation. Deduplication is typically done with a unique constraint on the event ID or booking ID in the receiving service’s database.

At-least-once with idempotent consumers is the standard contract in event-driven systems. Exactly-once is theoretically possible with additional coordination but practically rare – most systems accept at-least-once and invest in idempotency instead.

Change Data Capture as an Alternative Relay

Polling the outbox table works but adds load to the database. An alternative is Change Data Capture (CDC): instead of a polling process, you read directly from the database replication log.

Tools like Debezium connect to PostgreSQL’s logical replication stream or MySQL’s binlog and emit a change event for every row insert. Point Debezium at the outbox table and it publishes events automatically, with no polling query needed.

CDC-based relay has lower overhead and lower latency. The tradeoff: it’s another piece of infrastructure to run and operate. For smaller deployments, polling is simpler. For high-throughput systems where polling lag or database load is a concern, CDC is worth it.

Saga State Persistence

Related to the outbox is the question of where saga state itself lives.

The orchestrator driving the saga needs to know the current step, which steps have completed, and what the compensating actions are if something fails. Two common approaches:

Dedicated saga store: the orchestrator writes saga state to its own database table after each step. Simple, queryable, easy to inspect when debugging a stuck saga. The outbox pattern applies here too – saga state update and event publish happen in the same transaction.

Event sourcing: saga state is derived from the event log itself. There’s no explicit state table – you reconstruct the saga’s current position by replaying events. More complex to implement but gives you a full audit trail of everything that happened and an easy path to replaying failed sagas.

Most teams start with a dedicated saga store and move to event sourcing only if they need the audit trail or replay capability.

What Fails in Compensation

One more gap worth naming: compensating transactions can fail too.

You cancel the hotel reservation as part of rolling back a failed booking. The cancel call times out. Now your saga is stuck in a partially compensated state – the flight is cancelled but the hotel reservation is still live.

The standard answer is retry with backoff. Compensating transactions need to be idempotent for the same reason forward steps do. The outbox pattern helps here too – compensation requests go through the same outbox mechanism, so they’re durably enqueued even if the downstream service is temporarily unavailable.

If compensation keeps failing after retries, the saga is in a “saga failure” state that requires human intervention or a dead-letter queue. Building an operator UI for stuck sagas is not glamorous work, but it’s necessary in any production saga implementation.

The outbox pattern doesn’t make sagas simple. It removes the dual-write problem and gives you reliable event delivery – which turns a fundamentally broken approach into one that’s merely complex.

This is a cascading failure. One slow dependency turned into a full outage because the upstream service kept hammering it instead of stopping.

The Pattern

Circuit breakers fix this by failing fast. The name comes from electrical circuit breakers: when the current exceeds a threshold, the breaker trips and cuts power before anything burns. Same idea here.

The circuit has three states:

Closed – normal operation. Requests go through, failures are counted. When failures cross a configured threshold within a time window, the circuit trips.

Open – the circuit has tripped. Requests immediately return an error without calling the downstream service at all. No network round-trips, no timeouts, no extra load on a service that’s already struggling. After a cooldown period, the circuit moves to half-open.

Half-open – one test request is allowed through. If it succeeds, the circuit closes and normal operation resumes. If it fails, the circuit trips back to open and the cooldown resets.

The key insight is that sometimes the most useful thing you can do is stop trying. Retrying a failing service doesn’t help it recover. Backing off does.

Where You’ll Find It

Netflix Hystrix made this pattern famous, but it’s now table stakes in most infrastructure:

• Istio has it built into the service mesh at the sidecar level – no application code needed
• Resilience4j for Java
• Polly for .NET
• Most Redis clients ship with circuit breaking on connection failures

The mesh-level implementations are interesting because they work across languages and don’t require every service team to implement the pattern themselves. You configure thresholds in the mesh and it handles the state machine.

The Gotcha: Timeout Ordering

There’s one configuration mistake that makes circuit breakers useless: setting your request timeout longer than your circuit breaker’s failure window.

If your timeout is 30 seconds but your circuit breaker trips after 5 failures within 10 seconds – and each failure takes 25 seconds to time out – you’ll accumulate only one or two failures in any 10-second window. The circuit never trips because failures arrive too slowly to hit the threshold, even while the system is drowning in slow requests.

The rule: timeouts must be shorter than the failure window. If your breaker watches a 10-second window and trips at 5 failures, your per-request timeout needs to be well under 2 seconds. Otherwise the breaker is just decorative.

The Saga pattern is the alternative. You give up true atomicity in exchange for availability, scalability, and no cross-service locks.

How It Works

Break one big distributed transaction into a sequence of local transactions, each owned by a single service. Each step publishes an event or sends a message when it completes. If any step fails, run compensating transactions in reverse to undo the completed steps.

The classic example is a trip booking:

Step 1: Reserve flight    → local tx in Flight Service
Step 2: Reserve hotel     → local tx in Hotel Service
Step 3: Charge card       → local tx in Payment Service

If step 3 fails:

FORWARD:   [Flight ✓] → [Hotel ✓] → [Payment ✗]
ROLLBACK:              ← [Cancel Hotel] ← [Cancel Flight]

No global rollback. No coordinator holding locks across all three services. Each service knows how to undo its own step.

Two Flavors

Choreography – each service listens for events and decides what to do next based on what it hears. No central coordinator. Fully decoupled. Easy to start, and a debugging nightmare once you have more than a handful of services. Tracing a failed saga means reconstructing a sequence of events scattered across multiple event streams.

Orchestration – a central saga orchestrator drives the sequence. It tells each service what to do, collects results, and issues compensating calls on failure. Full visibility into saga state. Easier to reason about and monitor. The orchestrator becomes a coordination hub – not quite a single point of failure if you make it durable, but a complexity center. Most teams end up here in production.

The Tradeoff You Can’t Avoid

Sagas trade the “I” in ACID (isolation) for availability. While a saga is in-flight, intermediate state is visible to other parts of the system. A hotel room might be reserved even though the flight hasn’t been confirmed yet. Other services can observe that state.

You have to design for this explicitly:

• Eventual consistency – the system converges to a correct final state, but not atomically
• Idempotent compensating transactions – if your “cancel flight” call gets retried twice due to a network hiccup, it must be safe to run twice. Compensation that isn’t idempotent will make your failure recovery worse than the original failure.

Where It Shows Up

Basically every large-scale order or booking flow:

• Uber – ride booking, driver assignment, payment charge, all as separate service transactions
• Amazon – order placement, inventory reservation, fulfillment, shipping
• Netflix – subscription management, content licensing flows

These systems can’t afford to hold cross-service locks for the duration of a user-facing transaction. They accept that intermediate states exist and design their compensating logic carefully.

The saga pattern doesn’t make distributed transactions easy. It makes the tradeoffs explicit and gives you a way to build systems that stay available even when individual steps fail.