Multi-Paxos is the optimization that makes Paxos practical as a replicated log. The insight is that Phase 1 can be amortized across many slots once you have a stable leader.

What Phase 1 Actually Does

Recall basic Paxos. Phase 1 (Prepare/Promise) establishes that a proposer has the highest ballot number any acceptor will respond to. The acceptors promise to ignore any lower-numbered proposer. This is the “take control” phase – after it completes, the proposer has permission to drive proposals through Phase 2.

Phase 1 doesn’t say anything about which value gets proposed. It just establishes leadership over the acceptors for a given ballot number.

The Amortization

If the same leader runs Phase 1 once with a high ballot number, it doesn’t need to run Phase 1 again for subsequent log slots – as long as no other proposer challenges it with a higher ballot.

The leader runs Phase 1 once, gets promises from a majority, and then drives Phase 2 directly for slot 1, slot 2, slot 3, and so on. Each slot is just one round-trip (Accept/Accepted) instead of two. Throughput roughly doubles for steady-state operation under a stable leader.

Phase 1 (once):   Leader → Prepare(ballot=5) → Majority
                  Majority → Promise(ballot=5, ...) → Leader

Phase 2 (per slot):
  Slot 1:  Leader → Accept(ballot=5, slot=1, value=A) → Majority
           Majority → Accepted → Leader → Committed
  Slot 2:  Leader → Accept(ballot=5, slot=2, value=B) → Majority
           Majority → Accepted → Leader → Committed
  ...

This is Multi-Paxos. The Phase 1 cost is paid once per leader term, not once per log entry.

Leader Election

Basic Paxos has no notion of a leader – any node can propose at any time, which is why it’s correct but chaotic in practice. Multi-Paxos adds an informal leader election layer: one node is designated the distinguished proposer and the others voluntarily defer to it.

“Voluntarily” is the key word. If followers suspect the leader is dead (no heartbeats for a timeout period), any of them can run Phase 1 with a higher ballot to take over. The old leader doesn’t get a veto – it just gets its Accept messages rejected because the acceptors have now promised a higher ballot.

This is where Multi-Paxos implementations diverge significantly. The Paxos paper doesn’t specify how leader election works, what timeouts to use, or how to detect a live leader that’s just partitioned. Every real implementation makes different choices here, which is part of why Multi-Paxos implementations are hard to compare.

Log Holes

One complication basic Paxos avoids: with multiple slots running in parallel, you can get holes. The leader accepts slot 1 and slot 3 but slot 2 hasn’t been filled yet because a client request was lost. Followers can’t apply slot 3 to the state machine without knowing slot 2.

Multi-Paxos implementations handle this with a no-op: when a new leader wins an election, it runs Phase 1 for all uncommitted slots and fills any gaps with a no-op entry. This is what “leader reconciliation” looks like in practice – the new leader has to figure out what was committed before it took over and resolve any ambiguity.

Raft avoids holes entirely by requiring that the leader always send entries in order, with no gaps. This simplifies the implementation at the cost of some throughput (you can’t pipeline out-of-order).

Zab vs. Multi-Paxos

ZooKeeper uses Zab (ZooKeeper Atomic Broadcast), which is often described as “Paxos-like” but is actually closer to Multi-Paxos with some explicit differences.

The main one: Zab is designed around a primary-backup model where the primary sends updates in a strict FIFO order. The recovery protocol (Phase 1 equivalent in Zab) is designed to ensure the new primary has a superset of all committed entries before it starts serving. Zab also has an explicit synchronization phase where followers sync their logs to the new primary before it starts accepting writes.

Both tolerate the same number of failures and provide the same safety guarantees. The practical difference is that Zab is tuned for ZooKeeper’s specific write pattern (broadcast to all followers, wait for majority ack, apply in order) while Multi-Paxos is a more general framework.

Multi-Paxos vs. Raft

Multi-Paxos and Raft have the same fault tolerance and safety properties. The difference is that Raft specifies the things Multi-Paxos leaves up to the implementer – which is why Raft implementations tend to be more consistent across codebases.

Google’s Chubby uses a Paxos variant internally. Google Spanner uses a Paxos variant per shard. In both cases Google built their own leader election and recovery logic on top of the basic protocol – which is exactly the work Raft was designed to eliminate.

Log compaction is how you fix this. The idea is simple: once a prefix of the log has been applied to the state machine, you don’t need the individual entries anymore – just a snapshot of the state at that point.

Snapshots

A snapshot captures the complete state machine state at a given log index. Once the snapshot is written, all log entries up to and including that index can be discarded.

Before compaction:
  Log: [1: set x=1] [2: set y=2] [3: set x=3] [4: del y] [5: set z=1]
  Applied through index 4
  State machine: {x: 3, z: 1}

After compaction:
  Snapshot at index 4: {x: 3}  (z not in snapshot -- set z=1 not yet applied)
  Remaining log: [5: set z=1]

Wait – index 5 is retained because it hasn’t been applied yet when the snapshot is taken. The snapshot covers only committed, applied entries.

The snapshot metadata includes the last included index and last included term. These replace the log prefix. When AppendEntries includes a consistency check against the entry before the new one, the leader uses the snapshot’s last index and term as the baseline when the follower’s log has been compacted past that point.

When to Snapshot

Most implementations trigger a snapshot when the log grows past a size threshold – say 64MB or 256MB. The Raft paper leaves this as an implementation detail. etcd uses 100MB by default. Smaller thresholds mean more frequent snapshots and less log replay on restart, but snapshot creation has a cost.

Taking a snapshot of a large state machine can be expensive. If you’re snapshotting a database with gigabytes of data, the snapshot itself might take seconds. During that time the state machine is typically paused or copy-on-write semantics are used to take a consistent point-in-time snapshot without stopping serving. Most production implementations use copy-on-write (fork on Linux makes this cheap for process-based state machines).

InstallSnapshot RPC

Snapshots matter most for a new node joining the cluster or a node that has been down for a long time and fallen significantly behind.

If a follower is so far behind that the leader has already compacted the log entries it needs, AppendEntries won’t work – those entries are gone. Instead the leader sends the snapshot directly with InstallSnapshot RPC.

The follower receives the snapshot, writes it to stable storage, discards its entire existing log, and resets its state machine to the snapshot state. After that it can receive normal AppendEntries for the remaining log entries.

InstallSnapshot is also used when a brand new node joins the cluster. Rather than replaying potentially years of log history, the leader sends the current snapshot and the follower catches up from there.

etcd in Practice

etcd runs a snapshot at 100MB by default (configurable with --snapshot-count, which actually counts committed entries, not bytes – the default is 10,000 entries). After snapshotting, etcd retains the last 5 snapshots and the WAL (write-ahead log) segments needed to replay from the oldest retained snapshot. This gives you a window of history for debugging without storing the full log.

etcd also exposes the snapshot mechanism for operator use. etcdctl snapshot save takes a snapshot of the current state to a file. This is the recommended backup mechanism for etcd clusters – you back up the snapshot, not the raw data directory.

Leader Leases and Linearizable Reads

One related problem: reads in a basic Raft cluster go through the leader, which needs to confirm it’s still the leader before responding. The way to confirm is to send a heartbeat round to a majority before responding to a read – which is an extra round-trip per read, expensive under load.

Leader leases are the optimization. When a leader wins an election and sends its first heartbeats, it records the time and grants itself a lease for a fixed duration (typically the election timeout). During that window, it’s guaranteed to still be the leader – any new leader would need a full election timeout to emerge. So it can serve reads locally without a heartbeat round.

The caveat: lease-based reads depend on the leader’s clock being accurate relative to other nodes. If a leader’s clock runs fast and its lease hasn’t actually expired on other nodes’ clocks while it thinks it has, you can serve stale reads. etcd uses clock bounds and makes the lease duration conservative to avoid this. In practice, lease reads are the default in most etcd deployments because the latency benefit outweighs the theoretical risk on well-synchronized hardware clocks.

Raft’s log compaction is not exciting to implement but it’s essential for production deployments. A cluster without snapshot support will eventually fall over under its own log weight – or force every new node to replay history from the beginning, which is equally untenable.

This is the consensus problem. Paxos is the foundational answer. It’s also notoriously difficult to implement correctly – which is exactly why Raft exists. But you can’t really understand Raft without understanding what Paxos is doing and why.

The Constraint

Two-phase commit solves distributed atomicity but requires the coordinator to stay up. If the coordinator crashes after phase 1, participants are stuck holding locks, unable to proceed. That’s not consensus – that’s a single point of failure with extra steps.

Paxos removes that constraint. It guarantees agreement as long as a majority of nodes are up, regardless of which specific nodes fail. Any node can propose a value. No permanent coordinator required.

Roles

In Paxos, nodes play three roles – and in practice, any node can play any of them:

• Proposer – initiates a round, tries to get a value accepted
• Acceptor – votes on proposals, must follow the protocol strictly
• Learner – learns the final chosen value once consensus is reached

The Protocol

Paxos runs in two phases.

Phase 1 – Prepare / Promise

The proposer picks a proposal number n – unique, higher than any it has seen before – and sends PREPARE(n) to a quorum of acceptors (any majority).

Each acceptor that receives this:

• Promises to ignore any future PREPARE with a lower number
• If it has already accepted a value in a prior round, includes that (old_n, old_value) in its reply

Phase 2 – Accept / Accepted

If the proposer receives promises from a quorum:

• If any promise included a prior accepted value, the proposer must use the highest one – it cannot choose freely
• Otherwise it can propose any value it wants

It sends ACCEPT(n, value) to the quorum. Acceptors that haven’t made a conflicting promise write the value and reply ACCEPTED. Once a quorum accepts, the value is chosen.

Proposer       Acceptor A    Acceptor B    Acceptor C
   |-- PREPARE(5) -->|             |             |
   |-- PREPARE(5) ---|------------>|             |
   |<-- PROMISE(5) --|             |             |
   |<-- PROMISE(5) --|-------------|             |
   |-- ACCEPT(5,v) ->|             |             |
   |-- ACCEPT(5,v) --|------------>|             |
   |<-- ACCEPTED ----|             |             |
   |<-- ACCEPTED ----|-------------|             |
      [value v is now chosen]

Why It’s Correct

The safety guarantee – that no two values are ever chosen – comes from the quorum overlap. Any two majorities in an N-node cluster share at least one node. That shared node is the bridge: if a value was accepted in round n, a new proposer in round n+1 will hear about it from the shared node and be forced to continue it rather than propose something different.

The constraint that proposers must adopt the highest prior accepted value is what makes this work. A new round can never silently override an old one.

The Gotchas

Livelock. Two competing proposers can keep preempting each other with higher proposal numbers indefinitely – neither ever reaching phase 2. The fix is to elect a single distinguished proposer (a leader) and have everyone else defer to it. This is what Multi-Paxos does.

Single-value vs. Multi-Paxos. Basic Paxos agrees on one value. Real systems need a log of values – a sequence of commands. Multi-Paxos runs Paxos repeatedly for each log slot with a stable leader, amortizing the phase 1 cost across many entries. Once a leader is established, phase 1 only needs to run once per leader lifetime.

Implementation gap. The original paper was intentionally abstract. “Paxos Made Simple” is cleaner but still leaves enormous implementation latitude – enough that teams have shipped subtly broken versions. The gap between the protocol description and a correct, production-safe implementation is one of the main reasons Raft was designed as an explicit replacement.

Where It Runs

• Google Chubby – distributed lock service, underpins Bigtable and GFS coordination
• Apache ZooKeeper – uses Zab (ZooKeeper Atomic Broadcast), a Paxos variant
• Google Spanner – Paxos per shard for replication across datacenters

Paxos is the theory. If you’re building something new, you’ll probably implement Raft instead – same guarantees, actually debuggable. But the mental model of quorum-based two-phase consensus is worth having regardless of which one you reach for.

Raft was built as a direct response to that. Same safety guarantees, designed from scratch for understandability – the explicit goal was that a student should be able to implement it correctly from the paper alone.

It works. etcd runs on it. Kubernetes’ entire cluster state lives in etcd. CockroachDB uses a Raft group per range. TiKV, Consul – Raft is everywhere Paxos was supposed to be.

The Decomposition

Raft’s core move is breaking consensus into three subproblems that can be understood and implemented independently:

1. Leader election – pick one node as leader at any time
2. Log replication – leader accepts entries, replicates to followers
3. Safety – no two nodes ever commit conflicting entries

Everything flows through a single leader. There’s no “any node can propose” ambiguity like in basic Paxos. The leader is the sole source of truth for what enters the log. That constraint makes the rest of the protocol tractable.

Terms and Roles

Raft uses a logical clock called a term – a monotonically increasing integer. Each election starts a new term. If a node sees a message from a higher term than its own, it immediately updates and steps down if it was leader. Terms are how stale leaders get detected and evicted.

Every node is one of three things at any moment:

• Leader – accepts client requests, replicates entries, sends heartbeats to prevent new elections
• Follower – passive, responds to RPCs from leaders and candidates
• Candidate – transitional state during an election

Every node starts as a follower.

Leader Election

Followers expect periodic heartbeats from the leader via AppendEntries RPCs (sent even when there are no entries to replicate). If an election timeout fires without a heartbeat, the follower converts to a candidate:

1. Increment current term, vote for self
2. Send RequestVote to all peers
3. A peer grants its vote if it hasn’t voted this term and the candidate’s log is at least as up-to-date as its own
4. First candidate to collect a majority wins, becomes leader, starts sending heartbeats immediately

Split votes happen when two candidates start elections simultaneously and each gets half the cluster. Neither reaches majority. Both time out and retry with a new term. Raft handles this with randomized election timeouts (typically 150–300ms) – making simultaneous timeouts unlikely without coordination overhead.

Log Replication

Once a leader is elected:

1. Client sends a command to the leader
2. Leader appends the entry to its local log (not yet committed)
3. Leader sends AppendEntries to all followers in parallel
4. Once a majority have written the entry, the leader marks it committed
5. Leader applies the entry to its state machine, replies to client
6. On the next heartbeat, followers learn the commit index and apply the entry too

Client → [write x=5] → Leader
Leader  → AppendEntries(x=5) → Follower A  ✓
Leader  → AppendEntries(x=5) → Follower B  ✓
Leader  → AppendEntries(x=5) → Follower C  (unreachable)
[Majority = Leader + A + B → COMMIT] → apply x=5

Follower C is behind but not a problem. When it reconnects, the leader sends it the missing entries.

Safety: Log Matching

Raft’s key safety property: if two logs contain an entry with the same index and term, all preceding entries are identical.

This is enforced by the AppendEntries consistency check. When the leader sends an entry, it includes the index and term of the entry immediately before it. A follower rejects the append if that preceding entry doesn’t match its own log. The leader then backs up and resends from an earlier point until it finds a common prefix, then overwrites the diverged entries.

The vote restriction reinforces this: a node won’t vote for a candidate whose log is less up-to-date than its own. This guarantees the new leader always has all committed entries – it never needs to fetch missing entries from other nodes after winning an election.

Leader Failure

If the leader crashes, the election timeout fires on one of the followers and a new election starts. The vote restriction ensures the winner has the most complete log among the majority. No repair phase, no querying all nodes – safety comes from the constraint, not from post-election cleanup.

Raft vs. Paxos

Both tolerate up to (N-1)/2 failures in an N-node cluster. The fault tolerance is identical. The difference is that Raft implementations are actually correct in practice because the protocol is debuggable.

Where It Runs

• etcd – Kubernetes’ backing store; the entire cluster state (pods, services, configs) is a Raft log
• CockroachDB – Raft per range; each 512MB range is an independent 3–5 node Raft group
• TiKV – Raft groups per region, underpins TiDB
• Consul – service discovery and distributed KV

If you’re building anything that needs replicated state and strong consistency, Raft is the default answer. The paper is readable in an afternoon. The TLA+ spec exists if you want to verify your implementation. Start there.