CRDTs take a different position: design the data structure so that conflicts can never happen. Not “detect and resolve” – just structurally eliminate the problem.

The Core Idea

A CRDT is a data structure with a merge function that is commutative, associative, and idempotent. Those three properties together mean you can apply updates in any order, any number of times, from any subset of nodes, and always converge to the same state.

• Commutative: merge(A, B) = merge(B, A) – order doesn’t matter
• Associative: merge(A, merge(B, C)) = merge(merge(A, B), C) – grouping doesn’t matter
• Idempotent: merge(A, A) = A – applying the same update twice is harmless

If your data structure satisfies these, you can replicate it across nodes with no coordination. Each node applies updates locally and gossips state to peers. Eventually everyone has the same state.

The Simplest CRDT: G-Counter

A grow-only counter. Each node maintains a vector of counts, one slot per node. To increment, a node adds to its own slot. To read the total, sum all slots. To merge two replicas, take the element-wise max.

Node A: [3, 1, 0]   Node B: [2, 2, 0]
Merge:  [3, 2, 0]   -- take max per slot
Total:  5

Node A can never decrement B’s slot and Node B can never decrement A’s slot. Merges always move counters up, never down. Convergence is guaranteed.

Adding Decrements: PN-Counter

A grow-only counter can’t go down. If you need decrements, use two G-Counters – one for increments (P), one for decrements (N). The value is P - N.

Each node increments its P slot to add and its N slot to subtract. Merges take the element-wise max of both vectors. The final value is the sum of all P slots minus the sum of all N slots.

Sets: The Interesting Case

Sets are where CRDTs get non-obvious. The naive approach breaks immediately: if Node A removes an element at the same time Node B adds it, what’s the right answer?

G-Set (grow-only set) sidesteps this: elements can only be added, never removed. Merge is union. Works perfectly but is obviously limited.

2P-Set (two-phase set) adds a remove set R alongside the add set A. An element is in the set if it’s in A but not R. Once removed, it can never be re-added. Merge is union on both sets.

LWW-Element-Set (last-write-wins) attaches a timestamp to each add and remove. On conflict, the higher timestamp wins. Simple, but you’re back to trusting clocks.

OR-Set (observed-remove set) is the one that gets this right. Each add operation generates a unique tag. Remove only removes the specific tags that were observed at remove time. If Node A adds element X (tag: t1) and Node B concurrently adds X again (tag: t2), then Node A removing X (removing tag t1) leaves Node B’s addition (tag t2) intact. The element stays in the set.

This matches user intuition: you removed the copy you knew about, not the one that was being added concurrently somewhere else.

State-based vs. Operation-based

There are two families of CRDTs.

State-based (CvRDTs): nodes gossip their full state. Merge is applied on receipt. The merge function must be a join on a join-semilattice – a partial order where every pair of elements has a least upper bound. Simple to reason about, but shipping full state can be expensive for large structures.

Operation-based (CmRDTs): nodes broadcast operations instead of state. No merge function needed on state – instead, the transport layer must guarantee exactly-once delivery and causal ordering. More efficient, but the delivery guarantee is a real constraint.

In practice, libraries like Automerge and Yjs use operation-based CRDTs with a hybrid approach: operations for real-time sync, periodic state snapshots for catch-up after disconnection.

Where They’re Used

Collaborative editing is the obvious one. Google Docs uses OT (operational transformation), an older approach with similar goals. Figma uses CRDTs. Notion, Linear, and most newer collaborative tools have moved toward CRDTs because the math is cleaner and they handle offline edits naturally.

Distributed databases: Riak’s eventual consistency model was built around CRDTs. Redis has CRDT support in Redis Enterprise for multi-region deployments. Azure Cosmos DB uses them internally for its multi-master replication.

Shopping carts: Amazon’s Dynamo paper mentions the shopping cart as a canonical use case. Adds and removes to a cart from multiple devices or regions converge without a coordinator.

The Tradeoff

CRDTs don’t work for everything. The merge semantics have to match what your application actually means.

A bank account is not a PN-Counter. If your account shows $500 and you withdraw $200 concurrently from two ATMs, merging the two PN-Counters would show $100 – which is correct math but incorrect banking. The constraint “balance must not go negative” requires coordination. You can’t avoid that with a CRDT.

The rule: CRDTs work when the merge semantics match application intent and when the application can tolerate the conflict-free resolution behavior. They’re not a universal replacement for coordination – they’re a way to avoid coordination in cases where the math lines up.

Vector clocks give you the diagnosis. CRDTs give you a structural guarantee that the diagnosis is never needed.

Google Spanner runs distributed transactions across a globally replicated database at scale. It can’t afford to block on coordinator failure. Its solution is to not use a standalone coordinator at all – it runs 2PC on top of Paxos groups, which means the coordinator itself is fault-tolerant.

The Core Move: Paxos Groups as Participants

In classic 2PC, each participant is a single node. It votes, acquires locks, and waits. If it crashes, the transaction is stuck.

Spanner’s participants are Paxos groups. Each shard of data is managed by a Paxos group – typically 5 replicas spread across datacenters. The Paxos leader for each group acts as the 2PC participant. Lock acquisition and voting happen on the leader, and the decision is durably replicated to the group before the leader responds.

If the coordinator crashes, a new coordinator can be elected (it’s also a Paxos group). If a participant leader crashes, a new leader in that group takes over – the lock state and vote are already replicated, so the protocol can continue.

The blocking problem is solved not by changing the protocol but by making every node in the protocol fault-tolerant.

TrueTime

Spanner adds something 2PC doesn’t have: globally synchronized timestamps, accurate enough to use for transaction ordering.

Google runs GPS receivers and atomic clocks in each datacenter and exposes a TrueTime API that returns a time interval [earliest, latest] rather than a single timestamp. The interval accounts for clock uncertainty. The guarantee is that the real current time lies somewhere in that interval.

Spanner uses TrueTime to assign commit timestamps. The rule: a transaction’s commit timestamp must be after the latest bound of TrueTime at the moment the coordinator decides to commit. This means Spanner waits for a brief period – typically 1-7ms – before declaring a transaction committed, to ensure that any other transaction starting after this one will see a strictly later timestamp.

This is the “commit wait.” It’s intentional latency, paid to guarantee that commit timestamps accurately reflect real-world ordering.

External Consistency

The property this buys is external consistency: if transaction T1 commits before T2 starts (in real-world time), T2 will see T1’s writes. This is stronger than serializability – serializable transactions can be ordered in any way consistent with their conflicts, even if that reorders real-world causality. External consistency preserves real-world order.

In practical terms: if you commit a write and then tell a colleague “it’s committed, go read it,” their read will see your write. There’s no window where a causally later transaction misses a causally earlier committed write.

Most distributed databases don’t provide this. They provide serializability, which is strong, but allows committed writes to be temporarily invisible to transactions that started after the commit in real time. Spanner’s external consistency closes that gap.

What the 2PC Layer Actually Looks Like

A read-write transaction in Spanner:

1. Client reads data, buffers writes locally (no locks yet)
2. Client sends commit request to the coordinator Paxos group
3. Coordinator runs 2PC with each participant Paxos group:
   • Sends PREPARE to each participant leader
   • Each participant leader acquires locks, logs the prepare record to its Paxos group, replies YES
   • Coordinator picks a commit timestamp (TrueTime latest + epsilon), logs it to its group
   • Sends COMMIT with the timestamp to all participants
4. Coordinator waits out the commit wait interval
5. Participants apply the writes at the commit timestamp, release locks
6. Client sees commit confirmation

The 2PC messages are still there. The key difference is that every step that touches durable state goes through a Paxos replication round first. The coordinator can crash and be replaced. A participant leader can crash and its successor has the full prepare state. Nothing blocks permanently.

The Cost

The commit wait adds latency. For single-region transactions it’s small (sub-millisecond when clocks are well-synchronized). For multi-region transactions where the coordinator and participants are in different continents, round-trip latency dominates and the TrueTime uncertainty is larger – you’re looking at tens of milliseconds.

Spanner is designed for this tradeoff: strong global consistency in exchange for latency that scales with geographic distance. For financial systems, global inventory, or anything that requires real external consistency, that tradeoff is often correct. For low-latency local transactions, it’s usually not.

Classic 2PC is blocking because the coordinator is a single node. Spanner’s answer is not to fix 2PC – it’s to make every node in the 2PC protocol a replicated state machine.

Multi-Paxos is the optimization that makes Paxos practical as a replicated log. The insight is that Phase 1 can be amortized across many slots once you have a stable leader.

What Phase 1 Actually Does

Recall basic Paxos. Phase 1 (Prepare/Promise) establishes that a proposer has the highest ballot number any acceptor will respond to. The acceptors promise to ignore any lower-numbered proposer. This is the “take control” phase – after it completes, the proposer has permission to drive proposals through Phase 2.

Phase 1 doesn’t say anything about which value gets proposed. It just establishes leadership over the acceptors for a given ballot number.

The Amortization

If the same leader runs Phase 1 once with a high ballot number, it doesn’t need to run Phase 1 again for subsequent log slots – as long as no other proposer challenges it with a higher ballot.

The leader runs Phase 1 once, gets promises from a majority, and then drives Phase 2 directly for slot 1, slot 2, slot 3, and so on. Each slot is just one round-trip (Accept/Accepted) instead of two. Throughput roughly doubles for steady-state operation under a stable leader.

Phase 1 (once):   Leader → Prepare(ballot=5) → Majority
                  Majority → Promise(ballot=5, ...) → Leader

Phase 2 (per slot):
  Slot 1:  Leader → Accept(ballot=5, slot=1, value=A) → Majority
           Majority → Accepted → Leader → Committed
  Slot 2:  Leader → Accept(ballot=5, slot=2, value=B) → Majority
           Majority → Accepted → Leader → Committed
  ...

This is Multi-Paxos. The Phase 1 cost is paid once per leader term, not once per log entry.

Leader Election

Basic Paxos has no notion of a leader – any node can propose at any time, which is why it’s correct but chaotic in practice. Multi-Paxos adds an informal leader election layer: one node is designated the distinguished proposer and the others voluntarily defer to it.

“Voluntarily” is the key word. If followers suspect the leader is dead (no heartbeats for a timeout period), any of them can run Phase 1 with a higher ballot to take over. The old leader doesn’t get a veto – it just gets its Accept messages rejected because the acceptors have now promised a higher ballot.

This is where Multi-Paxos implementations diverge significantly. The Paxos paper doesn’t specify how leader election works, what timeouts to use, or how to detect a live leader that’s just partitioned. Every real implementation makes different choices here, which is part of why Multi-Paxos implementations are hard to compare.

Log Holes

One complication basic Paxos avoids: with multiple slots running in parallel, you can get holes. The leader accepts slot 1 and slot 3 but slot 2 hasn’t been filled yet because a client request was lost. Followers can’t apply slot 3 to the state machine without knowing slot 2.

Multi-Paxos implementations handle this with a no-op: when a new leader wins an election, it runs Phase 1 for all uncommitted slots and fills any gaps with a no-op entry. This is what “leader reconciliation” looks like in practice – the new leader has to figure out what was committed before it took over and resolve any ambiguity.

Raft avoids holes entirely by requiring that the leader always send entries in order, with no gaps. This simplifies the implementation at the cost of some throughput (you can’t pipeline out-of-order).

Zab vs. Multi-Paxos

ZooKeeper uses Zab (ZooKeeper Atomic Broadcast), which is often described as “Paxos-like” but is actually closer to Multi-Paxos with some explicit differences.

The main one: Zab is designed around a primary-backup model where the primary sends updates in a strict FIFO order. The recovery protocol (Phase 1 equivalent in Zab) is designed to ensure the new primary has a superset of all committed entries before it starts serving. Zab also has an explicit synchronization phase where followers sync their logs to the new primary before it starts accepting writes.

Both tolerate the same number of failures and provide the same safety guarantees. The practical difference is that Zab is tuned for ZooKeeper’s specific write pattern (broadcast to all followers, wait for majority ack, apply in order) while Multi-Paxos is a more general framework.

Multi-Paxos vs. Raft

Multi-Paxos and Raft have the same fault tolerance and safety properties. The difference is that Raft specifies the things Multi-Paxos leaves up to the implementer – which is why Raft implementations tend to be more consistent across codebases.

Google’s Chubby uses a Paxos variant internally. Google Spanner uses a Paxos variant per shard. In both cases Google built their own leader election and recovery logic on top of the basic protocol – which is exactly the work Raft was designed to eliminate.

Log compaction is how you fix this. The idea is simple: once a prefix of the log has been applied to the state machine, you don’t need the individual entries anymore – just a snapshot of the state at that point.

Snapshots

A snapshot captures the complete state machine state at a given log index. Once the snapshot is written, all log entries up to and including that index can be discarded.

Before compaction:
  Log: [1: set x=1] [2: set y=2] [3: set x=3] [4: del y] [5: set z=1]
  Applied through index 4
  State machine: {x: 3, z: 1}

After compaction:
  Snapshot at index 4: {x: 3}  (z not in snapshot -- set z=1 not yet applied)
  Remaining log: [5: set z=1]

Wait – index 5 is retained because it hasn’t been applied yet when the snapshot is taken. The snapshot covers only committed, applied entries.

The snapshot metadata includes the last included index and last included term. These replace the log prefix. When AppendEntries includes a consistency check against the entry before the new one, the leader uses the snapshot’s last index and term as the baseline when the follower’s log has been compacted past that point.

When to Snapshot

Most implementations trigger a snapshot when the log grows past a size threshold – say 64MB or 256MB. The Raft paper leaves this as an implementation detail. etcd uses 100MB by default. Smaller thresholds mean more frequent snapshots and less log replay on restart, but snapshot creation has a cost.

Taking a snapshot of a large state machine can be expensive. If you’re snapshotting a database with gigabytes of data, the snapshot itself might take seconds. During that time the state machine is typically paused or copy-on-write semantics are used to take a consistent point-in-time snapshot without stopping serving. Most production implementations use copy-on-write (fork on Linux makes this cheap for process-based state machines).

InstallSnapshot RPC

Snapshots matter most for a new node joining the cluster or a node that has been down for a long time and fallen significantly behind.

If a follower is so far behind that the leader has already compacted the log entries it needs, AppendEntries won’t work – those entries are gone. Instead the leader sends the snapshot directly with InstallSnapshot RPC.

The follower receives the snapshot, writes it to stable storage, discards its entire existing log, and resets its state machine to the snapshot state. After that it can receive normal AppendEntries for the remaining log entries.

InstallSnapshot is also used when a brand new node joins the cluster. Rather than replaying potentially years of log history, the leader sends the current snapshot and the follower catches up from there.

etcd in Practice

etcd runs a snapshot at 100MB by default (configurable with --snapshot-count, which actually counts committed entries, not bytes – the default is 10,000 entries). After snapshotting, etcd retains the last 5 snapshots and the WAL (write-ahead log) segments needed to replay from the oldest retained snapshot. This gives you a window of history for debugging without storing the full log.

etcd also exposes the snapshot mechanism for operator use. etcdctl snapshot save takes a snapshot of the current state to a file. This is the recommended backup mechanism for etcd clusters – you back up the snapshot, not the raw data directory.

Leader Leases and Linearizable Reads

One related problem: reads in a basic Raft cluster go through the leader, which needs to confirm it’s still the leader before responding. The way to confirm is to send a heartbeat round to a majority before responding to a read – which is an extra round-trip per read, expensive under load.

Leader leases are the optimization. When a leader wins an election and sends its first heartbeats, it records the time and grants itself a lease for a fixed duration (typically the election timeout). During that window, it’s guaranteed to still be the leader – any new leader would need a full election timeout to emerge. So it can serve reads locally without a heartbeat round.

The caveat: lease-based reads depend on the leader’s clock being accurate relative to other nodes. If a leader’s clock runs fast and its lease hasn’t actually expired on other nodes’ clocks while it thinks it has, you can serve stale reads. etcd uses clock bounds and makes the lease duration conservative to avoid this. In practice, lease reads are the default in most etcd deployments because the latency benefit outweighs the theoretical risk on well-synchronized hardware clocks.

Raft’s log compaction is not exciting to implement but it’s essential for production deployments. A cluster without snapshot support will eventually fall over under its own log weight – or force every new node to replay history from the beginning, which is equally untenable.

The name comes from ship design. A ship’s hull is divided into watertight compartments. One compartment floods, the others stay dry. The ship stays afloat. Without bulkheads, one breach sinks everything.

The Problem Without Bulkheads

Imagine a service that calls three downstream APIs – payment, inventory, and recommendations. You have a thread pool with 100 threads handling all outbound requests.

The recommendations API starts responding slowly. Requests pile up waiting. Threads get tied up waiting for responses that take 10 seconds instead of 100ms. Within minutes all 100 threads are occupied waiting on recommendations. Now payment requests and inventory requests can’t get a thread either. Your service is down for all three functions because one non-critical API got slow.

A circuit breaker would help here once the recommendations API is actually failing – but slow isn’t failing. The requests are completing, just slowly. The circuit never trips. The thread pool still drains.

The Fix: Separate Resource Pools

Partition your resources so that one downstream dependency can only consume its own allocation.

Instead of one pool of 100 threads, you have three pools:

• Payment: 40 threads
• Inventory: 40 threads
• Recommendations: 20 threads

The recommendations API goes slow. It consumes all 20 of its threads. Payment and inventory are unaffected – they have their own pools. The non-critical dependency is isolated in its own compartment.

Two Implementation Styles

Thread pool isolation is what Hystrix made popular. Each downstream call goes through a dedicated thread pool. Callers block on their own pool, not a shared one. Overhead: context switches between the caller thread and the pool thread for every call.

Semaphore isolation is lighter. Instead of separate thread pools, you put a semaphore in front of each dependency with a max concurrent count. Callers execute on their own thread but must acquire the semaphore first. No thread switching overhead. The tradeoff: if the call actually blocks the thread (synchronous I/O), you’re still tying up the caller’s thread. Better for fast calls, worse for slow ones.

For services making lots of downstream calls with variable latency, thread pool isolation is generally safer. The overhead is real but the isolation is stronger.

Combining with Circuit Breakers

Bulkheads and circuit breakers are complementary, not alternatives. They protect against different failure modes:

• Bulkhead: limits resource consumption from a slow or partially degraded dependency
• Circuit breaker: stops calls entirely when a dependency crosses a failure threshold

The typical pattern is both together. The bulkhead caps how many threads/connections one dependency can consume. The circuit breaker trips when that dependency starts returning errors. Between the two, you handle the full range of failure behavior from “slightly slow” to “completely down.”

Netflix Hystrix combined both in a single abstraction. Resilience4j does the same. In a service mesh like Istio, connection pool settings are the bulkhead and outlier detection is the circuit breaker – same ideas, configured at the mesh level rather than the application level.

What to Size the Pools At

This is the hard part. Too small and you’re rejecting valid traffic. Too large and the isolation breaks down.

A reasonable starting point: look at your normal concurrency for each dependency, add headroom for spikes, and set the max at roughly 2x your normal peak. Monitor pool utilization in steady state – if the recommendations pool is regularly at 18/20, you either need to size up or your recommendations calls are too slow.

Bulkheads don’t fix slow dependencies. They contain the blast radius while you do.

The outbox pattern closes that gap.

The Dual-Write Problem

Say the flight reservation step succeeds. You update your database to mark the reservation as confirmed and then publish a FlightReserved event to your message broker. These are two operations against two different systems. There’s no way to make them atomic without 2PC – which is exactly what sagas are trying to avoid.

If you write to the database first and then crash before publishing, the event is never delivered. The hotel reservation step never starts. The saga stalls. Depending on your monitoring, this might sit undetected for a while.

If you publish first and then crash before writing to the database, you’ve told the hotel step to proceed even though the flight reservation isn’t durably recorded. The flight service comes back up with no record of the reservation. You’ve introduced a phantom.

The Outbox

The fix: write the event to an outbox table in the same database, in the same local transaction as your state update. Then have a separate process relay events from the outbox to the message broker.

BEGIN TRANSACTION
  UPDATE flights SET status = 'reserved' WHERE id = ?
  INSERT INTO outbox (event_type, payload, created_at)
    VALUES ('FlightReserved', '{"booking_id": 123, ...}', NOW())
COMMIT

The state change and the event are now atomic with respect to the database. Either both are committed or neither is. The outbox row is durable.

A separate relay process – sometimes called the outbox processor or message relay – polls the outbox table, publishes each event to the message broker, and marks the row as processed once delivery is confirmed.

Relay process:
  SELECT * FROM outbox WHERE processed_at IS NULL ORDER BY created_at LIMIT 100
  FOR EACH row:
    publish to message broker
    UPDATE outbox SET processed_at = NOW() WHERE id = row.id

At-Least-Once Delivery

The relay delivers at-least-once. If the relay crashes after publishing but before marking the row as processed, it will publish the same event again on restart. Downstream consumers have to be idempotent – processing the same event twice should produce the same result as processing it once.

This is a real design constraint. Your hotel reservation step needs to check whether it already processed a given FlightReserved event before creating a new reservation. Deduplication is typically done with a unique constraint on the event ID or booking ID in the receiving service’s database.

At-least-once with idempotent consumers is the standard contract in event-driven systems. Exactly-once is theoretically possible with additional coordination but practically rare – most systems accept at-least-once and invest in idempotency instead.

Change Data Capture as an Alternative Relay

Polling the outbox table works but adds load to the database. An alternative is Change Data Capture (CDC): instead of a polling process, you read directly from the database replication log.

Tools like Debezium connect to PostgreSQL’s logical replication stream or MySQL’s binlog and emit a change event for every row insert. Point Debezium at the outbox table and it publishes events automatically, with no polling query needed.

CDC-based relay has lower overhead and lower latency. The tradeoff: it’s another piece of infrastructure to run and operate. For smaller deployments, polling is simpler. For high-throughput systems where polling lag or database load is a concern, CDC is worth it.

Saga State Persistence

Related to the outbox is the question of where saga state itself lives.

The orchestrator driving the saga needs to know the current step, which steps have completed, and what the compensating actions are if something fails. Two common approaches:

Dedicated saga store: the orchestrator writes saga state to its own database table after each step. Simple, queryable, easy to inspect when debugging a stuck saga. The outbox pattern applies here too – saga state update and event publish happen in the same transaction.

Event sourcing: saga state is derived from the event log itself. There’s no explicit state table – you reconstruct the saga’s current position by replaying events. More complex to implement but gives you a full audit trail of everything that happened and an easy path to replaying failed sagas.

Most teams start with a dedicated saga store and move to event sourcing only if they need the audit trail or replay capability.

What Fails in Compensation

One more gap worth naming: compensating transactions can fail too.

You cancel the hotel reservation as part of rolling back a failed booking. The cancel call times out. Now your saga is stuck in a partially compensated state – the flight is cancelled but the hotel reservation is still live.

The standard answer is retry with backoff. Compensating transactions need to be idempotent for the same reason forward steps do. The outbox pattern helps here too – compensation requests go through the same outbox mechanism, so they’re durably enqueued even if the downstream service is temporarily unavailable.

If compensation keeps failing after retries, the saga is in a “saga failure” state that requires human intervention or a dead-letter queue. Building an operator UI for stuck sagas is not glamorous work, but it’s necessary in any production saga implementation.

The outbox pattern doesn’t make sagas simple. It removes the dual-write problem and gives you reliable event delivery – which turns a fundamentally broken approach into one that’s merely complex.

This is a cascading failure. One slow dependency turned into a full outage because the upstream service kept hammering it instead of stopping.

The Pattern

Circuit breakers fix this by failing fast. The name comes from electrical circuit breakers: when the current exceeds a threshold, the breaker trips and cuts power before anything burns. Same idea here.

The circuit has three states:

Closed – normal operation. Requests go through, failures are counted. When failures cross a configured threshold within a time window, the circuit trips.

Open – the circuit has tripped. Requests immediately return an error without calling the downstream service at all. No network round-trips, no timeouts, no extra load on a service that’s already struggling. After a cooldown period, the circuit moves to half-open.

Half-open – one test request is allowed through. If it succeeds, the circuit closes and normal operation resumes. If it fails, the circuit trips back to open and the cooldown resets.

The key insight is that sometimes the most useful thing you can do is stop trying. Retrying a failing service doesn’t help it recover. Backing off does.

Where You’ll Find It

Netflix Hystrix made this pattern famous, but it’s now table stakes in most infrastructure:

• Istio has it built into the service mesh at the sidecar level – no application code needed
• Resilience4j for Java
• Polly for .NET
• Most Redis clients ship with circuit breaking on connection failures

The mesh-level implementations are interesting because they work across languages and don’t require every service team to implement the pattern themselves. You configure thresholds in the mesh and it handles the state machine.

The Gotcha: Timeout Ordering

There’s one configuration mistake that makes circuit breakers useless: setting your request timeout longer than your circuit breaker’s failure window.

If your timeout is 30 seconds but your circuit breaker trips after 5 failures within 10 seconds – and each failure takes 25 seconds to time out – you’ll accumulate only one or two failures in any 10-second window. The circuit never trips because failures arrive too slowly to hit the threshold, even while the system is drowning in slow requests.

The rule: timeouts must be shorter than the failure window. If your breaker watches a 10-second window and trips at 5 failures, your per-request timeout needs to be well under 2 seconds. Otherwise the breaker is just decorative.

When N changes to N+1 or N-1, nearly every key remaps to a different node. You go from a warm cache to a cold one in an instant – a thundering herd of cache misses hitting your database all at once. For a large cluster, that’s enough to take down a service.

Consistent hashing fixes this. Add or remove a node, and only the keys that were assigned to that node need to move. Everything else stays put.

The Hash Ring

Instead of hash(key) % N, arrange all possible hash values on a ring from 0 to 2^32. Each node gets a position on the ring by hashing its name or ID. To find which node owns a key: hash the key, then walk clockwise around the ring until you hit a node.

          0
         / \
    Node C   Node A
       |         |
    Node B --- Node D

Key K hashes to a point on the ring. The first node clockwise from that point owns K.

Now add a new node between B and C. Only the keys that were between B and the new node’s position need to move – from C to the new node. Everything else is unaffected. In an N-node cluster, adding one node moves roughly 1/N of the keys instead of nearly all of them.

Virtual Nodes

Raw consistent hashing has a load distribution problem. With a small number of physical nodes, the gaps between them on the ring are uneven – one node might own 30% of the keyspace, another 10%. Remove a node and all its load lands on a single neighbor.

Virtual nodes fix this: each physical node gets multiple positions on the ring, typically by hashing it with different suffixes (node1-0, node1-1, node1-2, …). With enough virtual nodes per physical node, the load distributes evenly across the ring. Removing one node scatters its keys across many neighbors rather than dumping them all on one.

This is the production configuration in every serious deployment.

Where It’s Used

• Amazon DynamoDB – consistent hashing across storage nodes, virtual nodes for even distribution
• Apache Cassandra – the token ring is consistent hashing; each node owns a range of tokens
• Memcached – the ketama client implements consistent hashing for cache node assignment
• Basically every distributed cache or storage system that needs to scale nodes without full reshuffling

The pattern generalizes beyond caching. Any system that needs to assign work or data to nodes – and occasionally add or remove nodes – runs into the same problem. Consistent hashing is the standard solution.

This is the consensus problem. Paxos is the foundational answer. It’s also notoriously difficult to implement correctly – which is exactly why Raft exists. But you can’t really understand Raft without understanding what Paxos is doing and why.

The Constraint

Two-phase commit solves distributed atomicity but requires the coordinator to stay up. If the coordinator crashes after phase 1, participants are stuck holding locks, unable to proceed. That’s not consensus – that’s a single point of failure with extra steps.

Paxos removes that constraint. It guarantees agreement as long as a majority of nodes are up, regardless of which specific nodes fail. Any node can propose a value. No permanent coordinator required.

Roles

In Paxos, nodes play three roles – and in practice, any node can play any of them:

• Proposer – initiates a round, tries to get a value accepted
• Acceptor – votes on proposals, must follow the protocol strictly
• Learner – learns the final chosen value once consensus is reached

The Protocol

Paxos runs in two phases.

Phase 1 – Prepare / Promise

The proposer picks a proposal number n – unique, higher than any it has seen before – and sends PREPARE(n) to a quorum of acceptors (any majority).

Each acceptor that receives this:

• Promises to ignore any future PREPARE with a lower number
• If it has already accepted a value in a prior round, includes that (old_n, old_value) in its reply

Phase 2 – Accept / Accepted

If the proposer receives promises from a quorum:

• If any promise included a prior accepted value, the proposer must use the highest one – it cannot choose freely
• Otherwise it can propose any value it wants

It sends ACCEPT(n, value) to the quorum. Acceptors that haven’t made a conflicting promise write the value and reply ACCEPTED. Once a quorum accepts, the value is chosen.

Proposer       Acceptor A    Acceptor B    Acceptor C
   |-- PREPARE(5) -->|             |             |
   |-- PREPARE(5) ---|------------>|             |
   |<-- PROMISE(5) --|             |             |
   |<-- PROMISE(5) --|-------------|             |
   |-- ACCEPT(5,v) ->|             |             |
   |-- ACCEPT(5,v) --|------------>|             |
   |<-- ACCEPTED ----|             |             |
   |<-- ACCEPTED ----|-------------|             |
      [value v is now chosen]

Why It’s Correct

The safety guarantee – that no two values are ever chosen – comes from the quorum overlap. Any two majorities in an N-node cluster share at least one node. That shared node is the bridge: if a value was accepted in round n, a new proposer in round n+1 will hear about it from the shared node and be forced to continue it rather than propose something different.

The constraint that proposers must adopt the highest prior accepted value is what makes this work. A new round can never silently override an old one.

The Gotchas

Livelock. Two competing proposers can keep preempting each other with higher proposal numbers indefinitely – neither ever reaching phase 2. The fix is to elect a single distinguished proposer (a leader) and have everyone else defer to it. This is what Multi-Paxos does.

Single-value vs. Multi-Paxos. Basic Paxos agrees on one value. Real systems need a log of values – a sequence of commands. Multi-Paxos runs Paxos repeatedly for each log slot with a stable leader, amortizing the phase 1 cost across many entries. Once a leader is established, phase 1 only needs to run once per leader lifetime.

Implementation gap. The original paper was intentionally abstract. “Paxos Made Simple” is cleaner but still leaves enormous implementation latitude – enough that teams have shipped subtly broken versions. The gap between the protocol description and a correct, production-safe implementation is one of the main reasons Raft was designed as an explicit replacement.

Where It Runs

• Google Chubby – distributed lock service, underpins Bigtable and GFS coordination
• Apache ZooKeeper – uses Zab (ZooKeeper Atomic Broadcast), a Paxos variant
• Google Spanner – Paxos per shard for replication across datacenters

Paxos is the theory. If you’re building something new, you’ll probably implement Raft instead – same guarantees, actually debuggable. But the mental model of quorum-based two-phase consensus is worth having regardless of which one you reach for.

Raft was built as a direct response to that. Same safety guarantees, designed from scratch for understandability – the explicit goal was that a student should be able to implement it correctly from the paper alone.

It works. etcd runs on it. Kubernetes’ entire cluster state lives in etcd. CockroachDB uses a Raft group per range. TiKV, Consul – Raft is everywhere Paxos was supposed to be.

The Decomposition

Raft’s core move is breaking consensus into three subproblems that can be understood and implemented independently:

1. Leader election – pick one node as leader at any time
2. Log replication – leader accepts entries, replicates to followers
3. Safety – no two nodes ever commit conflicting entries

Everything flows through a single leader. There’s no “any node can propose” ambiguity like in basic Paxos. The leader is the sole source of truth for what enters the log. That constraint makes the rest of the protocol tractable.

Terms and Roles

Raft uses a logical clock called a term – a monotonically increasing integer. Each election starts a new term. If a node sees a message from a higher term than its own, it immediately updates and steps down if it was leader. Terms are how stale leaders get detected and evicted.

Every node is one of three things at any moment:

• Leader – accepts client requests, replicates entries, sends heartbeats to prevent new elections
• Follower – passive, responds to RPCs from leaders and candidates
• Candidate – transitional state during an election

Every node starts as a follower.

Leader Election

Followers expect periodic heartbeats from the leader via AppendEntries RPCs (sent even when there are no entries to replicate). If an election timeout fires without a heartbeat, the follower converts to a candidate:

1. Increment current term, vote for self
2. Send RequestVote to all peers
3. A peer grants its vote if it hasn’t voted this term and the candidate’s log is at least as up-to-date as its own
4. First candidate to collect a majority wins, becomes leader, starts sending heartbeats immediately

Split votes happen when two candidates start elections simultaneously and each gets half the cluster. Neither reaches majority. Both time out and retry with a new term. Raft handles this with randomized election timeouts (typically 150–300ms) – making simultaneous timeouts unlikely without coordination overhead.

Log Replication

Once a leader is elected:

1. Client sends a command to the leader
2. Leader appends the entry to its local log (not yet committed)
3. Leader sends AppendEntries to all followers in parallel
4. Once a majority have written the entry, the leader marks it committed
5. Leader applies the entry to its state machine, replies to client
6. On the next heartbeat, followers learn the commit index and apply the entry too

Client → [write x=5] → Leader
Leader  → AppendEntries(x=5) → Follower A  ✓
Leader  → AppendEntries(x=5) → Follower B  ✓
Leader  → AppendEntries(x=5) → Follower C  (unreachable)
[Majority = Leader + A + B → COMMIT] → apply x=5

Follower C is behind but not a problem. When it reconnects, the leader sends it the missing entries.

Safety: Log Matching

Raft’s key safety property: if two logs contain an entry with the same index and term, all preceding entries are identical.

This is enforced by the AppendEntries consistency check. When the leader sends an entry, it includes the index and term of the entry immediately before it. A follower rejects the append if that preceding entry doesn’t match its own log. The leader then backs up and resends from an earlier point until it finds a common prefix, then overwrites the diverged entries.

The vote restriction reinforces this: a node won’t vote for a candidate whose log is less up-to-date than its own. This guarantees the new leader always has all committed entries – it never needs to fetch missing entries from other nodes after winning an election.

Leader Failure

If the leader crashes, the election timeout fires on one of the followers and a new election starts. The vote restriction ensures the winner has the most complete log among the majority. No repair phase, no querying all nodes – safety comes from the constraint, not from post-election cleanup.

Raft vs. Paxos

Both tolerate up to (N-1)/2 failures in an N-node cluster. The fault tolerance is identical. The difference is that Raft implementations are actually correct in practice because the protocol is debuggable.

Where It Runs

• etcd – Kubernetes’ backing store; the entire cluster state (pods, services, configs) is a Raft log
• CockroachDB – Raft per range; each 512MB range is an independent 3–5 node Raft group
• TiKV – Raft groups per region, underpins TiDB
• Consul – service discovery and distributed KV

If you’re building anything that needs replicated state and strong consistency, Raft is the default answer. The paper is readable in an afternoon. The TLA+ spec exists if you want to verify your implementation. Start there.

The Saga pattern is the alternative. You give up true atomicity in exchange for availability, scalability, and no cross-service locks.

How It Works

Break one big distributed transaction into a sequence of local transactions, each owned by a single service. Each step publishes an event or sends a message when it completes. If any step fails, run compensating transactions in reverse to undo the completed steps.

The classic example is a trip booking:

Step 1: Reserve flight    → local tx in Flight Service
Step 2: Reserve hotel     → local tx in Hotel Service
Step 3: Charge card       → local tx in Payment Service

If step 3 fails:

FORWARD:   [Flight ✓] → [Hotel ✓] → [Payment ✗]
ROLLBACK:              ← [Cancel Hotel] ← [Cancel Flight]

No global rollback. No coordinator holding locks across all three services. Each service knows how to undo its own step.

Two Flavors

Choreography – each service listens for events and decides what to do next based on what it hears. No central coordinator. Fully decoupled. Easy to start, and a debugging nightmare once you have more than a handful of services. Tracing a failed saga means reconstructing a sequence of events scattered across multiple event streams.

Orchestration – a central saga orchestrator drives the sequence. It tells each service what to do, collects results, and issues compensating calls on failure. Full visibility into saga state. Easier to reason about and monitor. The orchestrator becomes a coordination hub – not quite a single point of failure if you make it durable, but a complexity center. Most teams end up here in production.

The Tradeoff You Can’t Avoid

Sagas trade the “I” in ACID (isolation) for availability. While a saga is in-flight, intermediate state is visible to other parts of the system. A hotel room might be reserved even though the flight hasn’t been confirmed yet. Other services can observe that state.

You have to design for this explicitly:

• Eventual consistency – the system converges to a correct final state, but not atomically
• Idempotent compensating transactions – if your “cancel flight” call gets retried twice due to a network hiccup, it must be safe to run twice. Compensation that isn’t idempotent will make your failure recovery worse than the original failure.

Where It Shows Up

Basically every large-scale order or booking flow:

• Uber – ride booking, driver assignment, payment charge, all as separate service transactions
• Amazon – order placement, inventory reservation, fulfillment, shipping
• Netflix – subscription management, content licensing flows

These systems can’t afford to hold cross-service locks for the duration of a user-facing transaction. They accept that intermediate states exist and design their compensating logic carefully.

The saga pattern doesn’t make distributed transactions easy. It makes the tradeoffs explicit and gives you a way to build systems that stay available even when individual steps fail.

Two-phase commit (2PC) solves this. It’s been in production databases for decades. It also has a flaw that makes it unusable for anything long-running, which is why most modern large-scale systems don’t use it directly.

The Protocol

There’s a coordinator and a set of participants. The coordinator drives the protocol.

Phase 1 – Prepare

The coordinator sends PREPARE to every participant. Each participant:

• Writes the transaction to a durable log
• Acquires the necessary locks
• Replies YES (ready to commit) or NO (something failed)

Phase 2 – Commit or Abort

• If all participants said YES: coordinator sends COMMIT to everyone
• If anyone said NO: coordinator sends ABORT

Participants execute the decision and release their locks.

Coordinator         Node A        Node B
    |--- PREPARE ------->|             |
    |--- PREPARE ---------|----------->|
    |<-- YES ------------|             |
    |<-- YES ------------|-------------|
    |--- COMMIT -------->|             |
    |--- COMMIT ----------|----------->|

Clean path works fine. Every node ends up in the same state. Locks are held only for the duration of the protocol.

The Fatal Flaw

Crash the coordinator after phase 1 but before phase 2.

Participants are now stuck. They voted YES and locked their resources. They don’t know whether the coordinator decided to commit or abort. They can’t safely proceed in either direction – committing might be wrong if another participant said NO, aborting might be wrong if the coordinator was about to send COMMIT.

They wait. The locks stay held. The system is in limbo until the coordinator recovers.

This is the blocking problem. 2PC is a blocking protocol under failure. A single coordinator crash can stall the entire transaction indefinitely.

The Attempted Fix: 3PC

Three-phase commit inserts a pre-commit phase between prepare and commit:

PREPARE → PRE-COMMIT → COMMIT

Pre-commit lets participants infer the coordinator’s intent even if it crashes: if you received a pre-commit, the coordinator had seen all YES votes and was going to commit. You can safely commit yourself if the coordinator disappears after that point.

3PC eliminates most blocking scenarios. It costs one extra round of messages and still has edge cases under network partitions. It’s not widely used in practice – the complexity tradeoff isn’t worth it for most systems.

What Systems Actually Do

Most large-scale systems avoid 2PC for anything user-facing. Short, internal database transactions – a single service committing to one database – are fine. Cross-service transactions that could run for seconds or involve user interaction are not.

2PC is worth understanding because it makes the tradeoff concrete: you can have distributed atomicity, but you’re paying for it with availability under coordinator failure. Everything else in distributed transactions is a different point on that curve.

The question is: can you still tell whether one event caused another?

Vector clocks answer that question. They don’t give you wall-clock time. They give you something more useful: a precise definition of “happened before.”

How They Work

Each node maintains a vector of counters, one slot per node in the system. Call them [A, B, C] for a three-node cluster.

Three rules:

1. Local event – increment your own counter
2. Send a message – attach the current vector to the message
3. Receive a message – merge by taking the element-wise max of your vector and the received vector, then increment your own counter

A concrete trace:

Node A starts:  [A:0, B:0, C:0]
Node A event:   [A:1, B:0, C:0]
Node A sends to B, attaches [A:1, B:0, C:0]

Node B receives, merges: max([A:0,B:0,C:0], [A:1,B:0,C:0]) = [A:1,B:0,C:0]
Node B increments own:   [A:1, B:1, C:0]
Node B event:            [A:1, B:2, C:0]

What You Can Determine

Given two events with vectors V1 and V2, exactly one of three things is true:

The concurrent case is the important one. When neither vector dominates, the events genuinely have no causal relationship – they happened independently on different nodes without either one knowing about the other. No ordering can be inferred because no ordering exists.

Why This Matters

The alternative is last-write-wins: when two writes conflict, pick the one with the later timestamp and discard the other. This is simple to implement and silently destroys data. If two users edit the same document concurrently, you lose one of their changes without telling anyone.

Vector clocks let you detect that conflict honestly. DynamoDB and Riak use them (or variants) to surface conflicting writes to the application rather than silently resolving them. The application – or the user – can then merge the two versions intelligently.

“Happened-before” is a relationship you earn through the causal structure of the system, not one you assume from timestamps.

The Tradeoff

Vector size grows O(N) with the number of nodes. In a cluster with hundreds of nodes, attaching a full vector to every message gets expensive.

Dotted version vectors are a modern fix: they compress the vector by pruning entries that are causally dominated. Same core semantics, better space efficiency for large clusters.

Lamport clocks are the scalar predecessor – a single counter instead of a vector. They give you a consistent ordering (if A happened-before B, A’s timestamp is lower), but they don’t detect concurrency. Two events with different Lamport timestamps might still be concurrent. Vector clocks are strictly more informative.

Understanding Paxos is worth doing. It’s not just historical – Chubby, ZooKeeper, and Spanner are built on it or close variants, and many distributed systems interviews probe it. More importantly, every simplification Raft makes is a direct response to a specific pain point in Paxos. Understanding what Paxos gets wrong explains why Raft looks the way it does.

The Problem

Multiple nodes need to agree on a single value. The network is unreliable: messages can be dropped, reordered, or delayed arbitrarily. Nodes can crash at any time. There is no shared clock.

A minority of nodes can fail; the protocol must still make progress. The protocol must also be safe: once a value is chosen, no future round can choose a different value.

This is the consensus problem. Two-Phase Commit solves it only when the coordinator doesn’t fail. Paxos solves it even when any minority of nodes can fail, including the “coordinator.”

Roles

Any node can play any role (often all three simultaneously in practice):

• Proposer – initiates a round, trying to get a value agreed upon
• Acceptor – votes on proposals; the safety machinery lives here
• Learner – learns what value was finally chosen, acts on it

The Protocol – Two Phases

Phase 1: Prepare / Promise

1. The proposer picks a proposal number n. It must be unique across all proposers and higher than any proposal number the proposer has seen before. (Usually implemented with (timestamp, node_id) tuples or a monotonic counter stored durably.)

2. Sends PREPARE(n) to a quorum of acceptors (any majority).

3. Each acceptor that receives PREPARE(n):

   • If n is higher than any PREPARE it has responded to: replies PROMISE(n), committing to reject any future PREPARE or ACCEPT with a lower proposal number
   • If n is not higher: ignores or rejects the request
   • If it has already accepted a value in a prior round, it includes that (accepted_n, accepted_value) in the promise

Phase 2: Accept / Accepted

1. If the proposer receives PROMISE from a quorum:
   • If any promise included an (accepted_n, accepted_value), the proposer must use the value with the highest accepted_n – it cannot freely choose its own value
   • If no promise included an accepted value, the proposer can propose any value it wants
2. Sends ACCEPT(n, value) to the quorum
3. Each acceptor that receives ACCEPT(n, value):
   • If it hasn’t promised to ignore n: writes the value and replies ACCEPTED(n, value)
   • If it has made a promise with a higher number: ignores or rejects
4. Once a quorum of acceptors reply ACCEPTED with the same n → the value is chosen

Proposer          Acceptor A    Acceptor B    Acceptor C
   |-- PREPARE(5) -->|             |             |
   |-- PREPARE(5) ---|------------>|             |
   |<-- PROMISE(5) --|             |             |
   |<-- PROMISE(5) --|-------------|             |
   |-- ACCEPT(5,v) ->|             |             |
   |-- ACCEPT(5,v) --|------------>|             |
   |<-- ACCEPTED ---|              |             |
   |<-- ACCEPTED ---|--------------|             |
      [value v is chosen, Acceptor C was unreachable but quorum achieved]

Why It’s Correct

Safety (no two values can be chosen): Suppose value v was chosen at proposal n – meaning a quorum Q1 accepted it. Any later proposer n' > n that runs Phase 1 will receive promises from a quorum Q2. Q1 and Q2 must overlap (both are majorities). The overlapping acceptor already accepted v and includes (n, v) in its promise. The Phase 2 rule forces the new proposer to use v (since it’s the highest accepted value). So no different value can ever be chosen.

Liveness (the protocol makes progress): As long as a majority of nodes are reachable and a single stable proposer drives the protocol, a value will eventually be chosen. The protocol doesn’t guarantee progress if proposers keep competing – see the livelock problem below.

The Gotchas

Livelock with Competing Proposers

Two proposers P1 and P2 can keep preempting each other indefinitely:

P1 sends PREPARE(5), gets promises from quorum
P2 sends PREPARE(6) before P1 can reach ACCEPT -- quorum promises ignore n=5
P1's ACCEPT(5, v) is rejected -- quorum promised to ignore anything < 6
P1 increments to PREPARE(7), gets new promises
P2's ACCEPT(6, v) is now rejected
... loops forever

The fix is a distinguished proposer (leader): only one node proposes at a time. This is what Multi-Paxos formalizes – elect a stable leader, suppress all other proposers, and the livelock disappears. It’s not part of basic Paxos; it’s an add-on that every production implementation requires.

Single-Value vs. Log: Multi-Paxos

Basic Paxos agrees on one value. Real systems need to agree on a sequence of commands – a log. Multi-Paxos runs Paxos for each log slot, but with an optimization: once a stable leader exists, you can skip Phase 1 for subsequent slots. You pay the two-round-trip cost once per leader election, then only Phase 2 for every log entry.

The problem: Multi-Paxos is not a formal protocol. It’s described as an optimization on basic Paxos, and different implementations fill in the details differently. Log gaps are allowed – a slot can be accepted out of order. Cluster membership changes are not specified. Leader election is not specified. This leaves a large surface area for subtle bugs.

It’s Hard to Implement Correctly

“Paxos Made Simple” describes the algorithm clearly. Getting from there to a correct implementation in a production system requires handling: durability of acceptor state across crashes (promise/accept must be persisted before responding), leader election, log gap handling, log compaction, cluster membership changes, client request routing, and linearizable reads. None of these are in the paper.

Chubby’s authors wrote that their Paxos implementation took years to get right and required multiple rewrites. This is why Raft was created.

Paxos vs. Raft

The safety properties are equivalent. The differences are implementation surface:

Same fault tolerance: both tolerate ⌊(N-1)/2⌋ node failures.

Where Paxos Is Still Used

• Google Chubby – distributed lock service. Underpins Bigtable and GFS for leader election and metadata coordination.
• Apache ZooKeeper – uses Zab (ZooKeeper Atomic Broadcast), which is a Paxos variant with primary-order delivery guarantees added. Powers Kafka, HBase, and Hadoop ecosystem coordination.
• Google Spanner – Paxos per shard for cross-replica replication. Combined with TrueTime for external consistency.

New systems almost universally choose Raft. The guarantees are equivalent and the implementation surface is much smaller. Paxos matters because it’s what everything was built on, and because understanding what it leaves unspecified is the clearest way to understand why the distributed systems field spent a decade building on top of it before someone sat down and wrote a protocol that was complete enough to implement from scratch.

Why increasing maxPoolSize breaks your database. A deep dive into MongoDB connection pooling, SDAM, WiredTiger tickets, and preventing transaction pinning.

Introduction

In the MongoDB ecosystem, the “connection pool” is often treated as a set-and-forget configuration within MongoClient. However, when latency spikes or the database becomes unresponsive during high traffic, the pool is usually the first mechanism to break.

Unlike relational databases that often sit behind proxies like PgBouncer, MongoDB drivers are “smart”—they maintain direct connections to multiple nodes in a Replica Set or Sharded Cluster. This complexity means exhaustion isn’t just about running out of sockets; it’s about topology monitoring, transaction pinning, and the brutal reality of the thread-per-connection model on the server side.

How MongoDB Connects and Executes Queries

Before diving into pool exhaustion, it’s worth understanding the lifecycle of a MongoDB connection and query. The process is more involved than a simple socket open/close.

Connection Establishment

When your application calls MongoClient.connect(), the driver doesn’t immediately open a socket. Instead, it:

1. TCP Socket Creation: Establishes a TCP/IP connection to the MongoDB server (default port 27017). This is a standard three-way handshake at the network layer.

2. Wire Protocol Handshake: The client sends a hello command (formerly isMaster in older versions) to the server. This serves multiple purposes:

   • Verifies the server is a MongoDB instance
   • Retrieves server capabilities (wire protocol version, supported authentication mechanisms)
   • In MongoDB 4.4+, can include speculativeAuthenticate to start authentication concurrently, reducing latency

3. Authentication: If credentials are provided, the client authenticates using mechanisms like SCRAM (Salted Challenge Response Authentication Mechanism):

   • Client sends saslStart with chosen mechanism and initial payload
   • Server responds with a challenge
   • Client computes and sends proof derived from the challenge
   • Server verifies and grants access

This entire process—TCP handshake, TLS negotiation (if enabled), handshake, and authentication—typically takes 10-50ms depending on network latency and encryption overhead.

Query Execution Flow

Once authenticated, the connection is ready for operations. When you execute a query:

1. Connection Checkout: The driver checks out a connection from the pool (or waits if none are available—this is where exhaustion manifests).

2. Message Construction: The query is serialized into MongoDB’s wire protocol format. Modern MongoDB uses the OP_MSG opcode (introduced in 3.6), which includes:

   • Standard message header (request ID, response-to ID)
   • Flags indicating message properties
   • Sections containing the actual command/query data
   • Optional checksum for integrity

3. Network Transmission: The message is sent over the TCP socket to the server.

4. Server Processing: The mongod process:

   • Receives the message on a dedicated thread (in the thread-per-connection model)
   • Parses the query
   • Plans execution strategy
   • Accesses data through WiredTiger (subject to ticket limits)
   • Creates a cursor for read operations

5. Response: The server sends results back over the same connection. For large result sets, the driver uses cursors to fetch documents in batches (default batch size is 101 documents).

6. Connection Return: After the operation completes, the connection is returned to the pool for reuse—unless it’s pinned to a transaction (see “The Specific Pathology: Pinning” below).

Why This Matters for Pooling

Each of these steps has latency and resource costs. The connection establishment overhead (10-50ms) is why pooling exists: reusing connections avoids paying this cost for every operation. However, the server-side thread allocation means each connection has a real memory and scheduling cost on mongod, which is why blindly increasing pool size backfires.

Connection Pool Internals

Understanding how the pool manages connections internally is crucial for diagnosing exhaustion. The pool maintains connections in different states:

Connection States

1. Idle: Connections that are established and authenticated but not currently in use. These sit in the pool, ready for immediate checkout. They consume server-side resources (thread + memory) but are available for operations.

2. Checked Out: Connections actively executing an operation. When you call collection.find(), the driver checks out a connection, performs the operation, then returns it to idle—unless it’s pinned (see “The Specific Pathology: Pinning”).

3. Pending: Connections in the process of being established (TCP handshake, authentication). These don’t count toward maxPoolSize until fully established.

4. Closed: Connections that have been terminated, either due to errors, idle timeout, or explicit closure.

The Wait Queue

When all connections are checked out and the pool has reached maxPoolSize, new requests enter a wait queue. This queue has its own limits:

• waitQueueTimeoutMS: How long a request waits for a connection before throwing ConnectionPoolCheckoutFailed. Default is 0 (infinite wait—dangerous in production). Always set this to fail fast rather than hang indefinitely.

If the wait queue fills up, your application starts failing requests. This is often the first symptom of pool exhaustion, not the root cause.

Connection Lifecycle

The connection lifecycle follows these states and transitions:

[Not Created]
    │
    │ connect()
    ▼
[Pending] ── TCP handshake, authentication ──► [Idle]
    │                                              │
    │                                              │ Driver request
    │                                              ▼
    │                                        [Checked Out]
    │                                              │
    │                                              │ Operation complete
    │                                              │
    │                                              ▼
    │                                          [Idle] ◄──┐
    │                                              │     │
    │                                              │     │ Return to pool
    │                                              │     │
    │                                              │     │
    │                                              │ startTransaction()
    │                                              ▼     │
    │                                          [Pinned] ──┘
    │                                              │
    │                                              │ ʕ•ᴥ•ʔ DANGEROUS:
    │                                              │ Exclusive to session
    │                                              │ until commit/abort
    │                                              │
    │                                              │ commit/abortTransaction()
    │                                              ▼
    │                                          [Idle]
    │                                              │
    │                                              │ maxIdleTimeMS reached
    │                                              ▼
    │                                          [Closed]
    │                                              │
    └──────────────────────────────────────────────┘

State Descriptions:

• Pending: Connection is being established (TCP handshake, authentication)
• Idle: Connection is established and available in the pool for immediate use
• Checked Out: Connection is actively executing an operation
• Pinned: Connection is reserved for a transaction (removed from pool until transaction completes)
• Closed: Connection has been terminated

The pool lazily creates connections up to maxPoolSize. If minPoolSize > 0, it pre-warms that many connections at startup. Connections idle longer than maxIdleTimeMS are closed to free server resources.

SDAM: The Hidden Connection Overhead

MongoDB drivers implement SDAM (Server Discovery and Monitoring), which maintains connections to all nodes in a Replica Set or Sharded Cluster, not just the primary.

How SDAM Works

1. Initial Discovery: On MongoClient.connect(), the driver connects to seed addresses and sends hello commands to discover all replica set members.

2. Continuous Monitoring: Every heartbeatFrequencyMS (default 10 seconds), the driver sends hello to each known node to:

   • Detect role changes (primary elections, step-downs)
   • Check server capabilities
   • Update topology description

3. Topology Events: When a primary election occurs, the driver:

   • Closes connections to the old primary
   • Opens connections to the new primary
   • Potentially drains and recreates pools

The Connection Multiplier

If you have a 3-node replica set and set maxPoolSize=50, you’re not creating 50 connections—you’re creating up to 150 connections (50 per node). SDAM maintains separate pools for each server in the topology.

During a primary election, the driver may temporarily exceed this as it transitions between nodes. If you’re running 20 app instances, that’s 3,000 potential connections during failover, even if only one node is serving traffic.

Takeaway: Your maxPoolSize is per-server, not per-client. In replica sets, multiply by the number of nodes you’re monitoring.

Connection Pool Configuration Options

Here are the key options that affect pool behavior:

• maxPoolSize: Maximum connections per server. Default 100. This is the most commonly misconfigured setting.

• minPoolSize: Minimum idle connections to maintain. Default 0. Setting this > 0 pre-warms the pool but consumes server resources even when idle.

• maxIdleTimeMS: Close idle connections after this duration. Default 0 (never close). Useful in serverless where you want aggressive cleanup.

• waitQueueTimeoutMS: How long to wait for a connection before failing. Default 0 (wait forever—dangerous in production). Always set this to fail fast rather than hang indefinitely.

• connectTimeoutMS: Timeout for initial connection establishment. Default 30 seconds.

• socketTimeoutMS: Timeout for individual operations. Default 0 (no timeout). Different from connection checkout timeout.

• heartbeatFrequencyMS: How often SDAM checks server status. Default 10 seconds. Lower values detect failovers faster but increase network overhead.

• TCP Keepalive: Often overlooked. If your app sits behind a firewall or load balancer (e.g., AWS Network Load Balancer) with a 350s idle timeout, and your pool has maxIdleTimeMS > 350s, the LB will silently drop the connection. The driver won’t know until it tries to write to the socket and fails. Important: socketTimeoutMS does not send keepalive probes. Best Practice: Set maxIdleTimeMS to be slightly lower than your infrastructure’s timeout (e.g., 120000ms). On Linux, also tune net.ipv4.tcp_keepalive_time to be less than the cloud provider’s timeout (usually < 120 seconds).

Example Configuration (Node.js):

1const client = new MongoClient(uri, {
2  maxPoolSize: 10,              // Small pool for 16-core DB
3  minPoolSize: 2,               // Pre-warm 2 connections
4  maxIdleTimeMS: 30000,         // Close idle after 30s
5  waitQueueTimeoutMS: 5000,     // Fail after 5s wait
6  connectTimeoutMS: 10000,      // 10s connection timeout
7  socketTimeoutMS: 45000,       // 45s operation timeout
8  heartbeatFrequencyMS: 10000   // Check every 10s
9});

The Cost of a Connection in mongod

While MongoDB has moved toward a more asynchronous networking model in recent versions, historically (and effectively, for resource planning), it relies on a synchronous thread-per-connection model.

When your application opens a new connection:

1. Network: TCP handshake + TLS negotiation (CPU intensive).
2. Authentication: SCRAM or X.509 handshakes occur.
3. Memory: The mongod process allocates stack memory (typically ~1MB) for the thread handling that connection.
4. Context: The internal serviceExecutor must schedule this thread.

If you set maxPoolSize=100 on 50 Kubernetes pods, you are allowing 5,000 potential concurrent threads on the Primary. Before you hit hardware limits, you will likely hit WiredTiger ticket exhaustion.

The Math: Sizing for WiredTiger

The naive approach is to increase maxPoolSize when you see ConnectionPoolCheckoutFailed errors. This is usually wrong.

WiredTiger (the storage engine) uses “tickets” to control concurrency. In MongoDB versions prior to 7.0, there are 128 read tickets and 128 write tickets by default. MongoDB 7.0+ uses dynamic ticketing that adjusts based on workload, but still caps at 128 per type. Note on MongoDB 7.0+: While the hard cap remains 128 tickets, the dynamic algorithm often lowers this limit (e.g., to 60 or 80) in real-time to preserve latency. This means you might see queuing before you hit 128 concurrent active queries. If 1,000 connections try to read simultaneously, 872 of them are just queuing inside the kernel, consuming RAM but doing zero work.

The optimal pool size formula remains tied to hardware capability, not traffic volume:

If your database has 16 cores and you have 20 application instances, a maxPoolSize of 100 is overkill. You might actually achieve higher throughput with a pool size of 2 or 3 per instance, preventing the database from spending all its cycles on context switching.

Note: The (Cores * 2) rule applies strictly to the active executing threads. You can have a larger pool to handle network jitter, but waitQueueTimeoutMS is your safety valve. If you need 100 connections to saturate the CPU, your queries are likely too slow (unindexed). Fix the queries, don’t inflate the pool.

Workload Considerations:

• CPU-Bound Workloads: The formula above applies directly. Each active connection should map to an executing thread.
• IO-Bound Workloads: If the working set doesn’t fit in RAM and disk I/O is the bottleneck, a slightly larger pool might help hide I/O latency, though typically minPoolSize is more relevant here to avoid establishing connections during I/O spikes.
• Atlas Proxy / Serverless: If using MongoDB Atlas Serverless or the Atlas Proxy, the connection limits are handled differently (via a proxy layer), and the “100 limit” on the client side is less about server stress and more about client-side throttling.

The Specific Pathology: “Pinning”

Since you are versed in Mongo, you know the drivers implement the SDAM (Server Discovery and Monitoring) specification. But the most dangerous feature regarding pooling is Client-Side Operation Pinning, specifically with Transactions.

In a standard read/write, the driver grabs a connection, executes, and returns it. In a Transaction (Multi-Document ACID):

1. session.startTransaction() is called.
2. The driver pins a connection to that session.
3. That connection is removed from the pool and reserved exclusively for this session until commitTransaction or abortTransaction is called.

The Exhaustion Scenario

If you have maxPoolSize=50 and an API endpoint that takes 2 seconds to execute some logic inside a transaction, you can only serve 25 requests per second. The 26th request will wait. If the logic hangs (e.g., waiting for a third-party HTTP call), you will drain the pool instantly, causing a cascading failure across the app.

ʕ•ᴥ•ʔ Critical Rule: Never perform network I/O or long computations inside a MongoDB transaction. Transactions should be atomic, fast, and only contain database operations.

Practical Examples: The Right Way vs. The Wrong Way

Proper Client Initialization

Node.js - Express (Correct):

 1// app.js - Initialize once at startup
 2const { MongoClient } = require('mongodb');
 3
 4let client;
 5let db;
 6
 7async function initDatabase() {
 8  client = new MongoClient(uri, {
 9    maxPoolSize: 10,
10    waitQueueTimeoutMS: 5000
11  });
12  await client.connect();
13  db = client.db('myapp');
14  console.log('Database connected');
15}
16
17// Initialize before starting server
18initDatabase().then(() => {
19  app.listen(3000);
20});
21
22// Use the same client instance everywhere
23app.get('/users', async (req, res) => {
24  const users = await db.collection('users').find({}).toArray();
25  res.json(users);
26});

Node.js - Express (Wrong - Creates New Pool Per Request):

1// ʕ•̀ω•́ʔ DON'T DO THIS
2app.get('/users', async (req, res) => {
3  const client = new MongoClient(uri);  // New pool every request!
4  await client.connect();
5  const db = client.db('myapp');
6  const users = await db.collection('users').find({}).toArray();
7  await client.close();  // Closes pool, but too late
8  res.json(users);
9});

Python - Gunicorn (Correct):

 1# app.py - Create client after fork
 2from pymongo import MongoClient
 3
 4client = None
 5db = None
 6
 7def on_starting(server):
 8    # This runs in the master process
 9    pass
10
11def when_ready(server):
12    # This runs in each worker after fork
13    global client, db
14    client = MongoClient(uri, maxPoolSize=10)
15    db = client['myapp']
16
17@app.route('/users')
18def get_users():
19    users = list(db.users.find({}))
20    return jsonify(users)

Python - Gunicorn (Wrong - Fork-Unsafe):

 1# ʕ•̀ω•́ʔ DON'T DO THIS
 2from pymongo import MongoClient
 3
 4# Created in master process - NOT fork-safe!
 5client = MongoClient(uri)
 6db = client['myapp']
 7
 8@app.route('/users')
 9def get_users():
10    # Child processes share parent's file descriptors = deadlocks
11    # This often results in AutoReconnect errors or 
12    # 'file descriptor bad' errors because the socket 
13    # is shared across memory boundaries
14    users = list(db.users.find({}))
15    return jsonify(users)

Java - Spring Boot:

 1spring:
 2  data:
 3    mongodb:
 4      uri: "mongodb://user:pass@host1:27017,host2:27017/?replicaSet=myRS&maxPoolSize=10&minPoolSize=2&maxIdleTimeMS=30000"
 5
 6// Or if using a Configurer Bean (safe way to customize without breaking auto-config)
 7@Configuration
 8public class MongoConfig {
 9    @Bean
10    public MongoClientSettingsBuilderCustomizer customizer() {
11        return builder -> builder.applyToConnectionPoolSettings(pool -> {
12            pool.maxSize(10);
13            pool.maxWaitTime(5, TimeUnit.SECONDS);
14        });
15    }
16}

The Health Check Trap:

ʕ•ᴥ•ʔ Warning: Do not create a new MongoClient inside your Kubernetes readiness/liveness probe. This is a classic DOS attack on your own database. The probe runs every 10s; if it creates a new connection each time without proper closure, you will exhaust the server limits in minutes. Use the singleton client to run a lightweight command like db.command({ ping: 1 }).

Monitoring Pool Metrics

Node.js - Command Monitoring:

 1const { MongoClient } = require('mongodb');
 2
 3const client = new MongoClient(uri, {
 4  monitorCommands: true  // Enable command monitoring
 5});
 6
 7client.on('commandStarted', (event) => {
 8  console.log('Command:', event.commandName);
 9});
10
11client.on('commandSucceeded', (event) => {
12  console.log('Duration:', event.duration, 'ms');
13});
14
15// For connection pool metrics, use pool monitoring events
16client.on('connectionCheckedOut', (event) => {
17  if (event.duration > 100) {  // Checkout took > 100ms
18    console.warn('Slow connection checkout:', event.duration, 'ms');
19  }
20});
21
22client.on('commandFailed', (event) => {
23  if (event.error.code === 'ConnectionPoolCheckoutFailed') {
24    console.error('Pool exhausted!');
25  }
26});

Go - Pool Monitoring:

 1import (
 2    "go.mongodb.org/mongo-driver/event"
 3    "go.mongodb.org/mongo-driver/mongo"
 4)
 5
 6poolMonitor := &event.PoolMonitor{
 7    Event: func(evt *event.PoolEvent) {
 8        switch evt.Type {
 9        case event.ConnectionCreated:
10            fmt.Println("Connection created")
11        case event.ConnectionCheckedOut:
12            fmt.Println("Connection checked out")
13        case event.ConnectionCheckoutFailed:
14            fmt.Println("Checkout failed:", evt.Reason)
15        case event.ConnectionReturned:
16            fmt.Println("Connection returned")
17        }
18    },
19}
20
21clientOptions := options.Client().
22    ApplyURI(uri).
23    SetPoolMonitor(poolMonitor).
24    SetMaxPoolSize(10)
25
26client, _ := mongo.Connect(context.TODO(), clientOptions)

Driver Nuances & Pitfalls

Node.js (Mongoose / Native Driver)

The Node driver is asynchronous, but the pool logic is strict.

• The Singleton Mistake: A common pattern in serverless or improper Express setup is calling MongoClient.connect() inside the route handler. This creates a new pool for every request, rapidly hitting the Atlas connection limit (e.g., 1,500 connections on M10). Always initialize the client once at startup.
• Unified Topology: The Node.js driver (v4.0+) uses Unified Topology by default (it’s the only option). This keeps a background monitoring thread that can detect step-downs. If your pool churns during a primary election, check if your heartbeatFrequencyMS is tuned correctly.

Python (PyMongo)

• Fork Safety: MongoClient is not fork-safe. If you use gunicorn or uwsgi with forking, you must create the client after the fork (e.g., inside the worker process initialization), or you will end up with child processes trying to use the parent’s file descriptors, leading to deadlocks and obscure socket errors. This often manifests as AutoReconnect errors or “file descriptor bad” errors.
• Wait Queue: In PyMongo 4.0+, waitQueueMultiple was removed. Use waitQueueTimeoutMS to control how long threads wait for connections. Limit your application’s thread pool size to control concurrency.

Java (Spring Boot / MongoDB Java Driver)

• Automatic Pooling: The MongoTemplate handles pooling automatically. Ensure spring.data.mongodb.database settings do not override defaults with aggressive timeouts. The Java driver’s default maxPoolSize is 100, which is often too high for production.

Go (mongo-go-driver)

• The Go driver respects context. If your Connect or Find context times out, the connection might be closed or returned to the pool depending on the stage.
• Monitor PoolEvent via the SetPoolMonitor option to visualize exactly when connections are created vs. checked out.

Serverless & Atlas Implications

In MongoDB Atlas, connection limits are hard tiers.

• M0 (Free Tier): 500 connections per node
• M10: 1,500 connections per node
• M20: 3,000 connections per node
• Serverless Instances: These bill by “Read Processing Units” but still require connection management.

If you are running on AWS Lambda or Vercel, you cannot maintain a persistent pool. Every “warm” container might keep a pool of 10 open. 100 concurrent Lambdas = 1,000 connections. Solution: You typically cannot use standard pooling here. You must aggressively close connections (set maxIdleTimeMS very low) or use the Atlas Data API (HTTP based) instead of the TCP driver to avoid socket exhaustion.

Monitoring & Diagnosis

How do you know it’s a pool issue and not a slow query?

1. Check the Server:

   1db.serverStatus().connections
   2// look for { "current": 1200, "available": 50 }
   

   If current is high, your app is leaking connections or scaling too wide.

2. Check the Queue: Monitor WiredTiger ticket usage via db.serverStatus().wiredTiger.concurrentTransactions or connection counts. If active connections significantly exceed your pool size or WiredTiger tickets are exhausted, your latency is due to queuing, not network speed.

3. Zombie Connections: When a client crashes hard (OOM Kill) without sending a TCP FIN, the server keeps the connection open until the server-side tcp_keepalive times out. In high-churn Kubernetes environments, this can exhaust the server’s file descriptors even if the active pool count seems low. This reinforces the need for aggressive maxIdleTimeMS and proper TCP keepalive configuration.

4. Driver Metrics: Enable command monitoring in your driver. If connectionCheckoutTime spikes while commandDuration (server execution time) stays low, the problem is client-side exhaustion. The database is fast; the line to get to it is slow.

5. Connection Churn Rate: Monitor the rate of new connections (connectionsCreated per second). If this is high (>10/sec per node) while your total pool count is stable, your maxIdleTimeMS is likely too low. You are burning server CPU on SSL handshakes rather than reusing connections. Check db.serverStatus().connections.created and compare it to your connection pool turnover rate.

Conclusion

In MongoDB, “Connection Pool Exhaustion” is often a misnomer for “Transaction Pinning” or “Overscaling.” Because the drivers are complex state machines interacting with Replica Sets, simply bumping maxPoolSize masks the problem until it crashes mongod via memory pressure or ticket exhaustion.

The fix is almost always:

1. Lower the pool size to match hardware (threads <= cores).
2. Shorten transaction scopes to microseconds.
3. Treat the MongoClient as a sacred singleton.

motivation

I was recently discussing with my CTO about databases. There was one point where we were thinking about the CAP theorem. Then he randomly mentioned that MySQL is not CA. I was surprised because I always thought that MySQL is a CA database at its core.

Now I had to search and understand why MySQL is not CA.

CAP theorem

We have always been taught in Engineering that CAP is one of the most important theorems in distributed systems and databases. We just accepted it. Time to question and learn ┏ʕ •ᴥ•ʔ┛

Here is what it actually means.

Imagine a bank server, when you check your account balance it should not randomly change it based on the server you are hitting and it should be consistent across all nodes. But in a distributed system, network failures (partitions) can occur. The CAP theorem states that you can only guarantee 2 out of 3 properties:

• Consistency (C): All nodes see the same data at the same time
• Availability (A): Every request receives a response, even if it’s not the most recent data
• Partition Tolerance (P): The system continues to operate despite network partitions

Here are practical examples of each combination:

CA Systems (Consistent + Available, but not Partition Tolerant)

Example: Traditional Relational Databases (like PostgresSQL in single-node setups)

Imagine a single PostgresSQL database server in a bank. When you check your balance:

• Consistency: Every time you query your balance, you get the exact same amount ($100.00)
• Availability: The system always responds to your requests
• Partition Tolerance: If the network connection to the database fails, you can’t access your account

This works perfectly in non-distributed environments, but fails when you need to scale across multiple data centers.

CP Systems (Consistent + Partition Tolerant, but sacrifices Availability)

Example: Apache ZooKeeper or etcd

Imagine a distributed configuration management system like ZooKeeper:

• Consistency: All nodes agree on configuration values (like which server is the leader)
• Partition Tolerance: If some nodes lose network connectivity, the remaining nodes continue operating
• Availability: During a network partition, some requests might be blocked to maintain consistency

In a ZooKeeper cluster, if a majority of nodes become unreachable, the system stops accepting writes to prevent inconsistent states.

AP Systems (Available + Partition Tolerant, but sacrifices Consistency)

Example: Amazon DynamoDB or Cassandra

Imagine a social media platform’s like counter system:

• Availability: The system always responds to requests, even during network failures
• Partition Tolerance: The system continues operating across multiple data centers despite network issues
• Consistency: During partitions, you might see different like counts on different servers

You might see a post with 100 likes on one page refresh, then 102 likes on the next refresh, as the systems reconcile the differences asynchronously.

Why MySQL is not CA

While MySQL can achieve CA in single-node deployments, it cannot be truly CA in distributed scenarios. When you add MySQL replicas or clustering solutions like MySQL Group Replication:

• During network partitions, MySQL must choose between consistency (rejecting writes) or availability (allowing potentially conflicting writes)
• MySQL’s synchronous replication can become unavailable during partitions if strict consistency is enforced
• Traditional asynchronous replication trades consistency for availability (so it becomes AP, not CA). Even with semi-sync replication, MySQL typically prioritizes availability over strict consistency—clients may see stale reads.

It doesn’t enforce strong ACID guarantees across replicas the way PostgreSQL doe

Why PostgreSQL is considered CA

Postgres in standalone mode: Definitely CA (strong ACID, always responds if node is up).

In replication setups:

PostgreSQL replication (especially with synchronous replication) prioritizes consistency first. A commit can be acknowledged only after replicas confirm the write (if you configure synchronous replication).

This means if replicas aren’t reachable, Postgres will prefer not to serve stale data (sacrificing availability to preserve consistency).

So Postgres is effectively CA (or CP in distributed setups).

This is why distributed databases like CockroachDB or TiDB are often preferred for truly distributed, partition-tolerant systems.

ʕ ● ᴥ ●ʔ