CRDTs take a different position: design the data structure so that conflicts can never happen. Not “detect and resolve” – just structurally eliminate the problem.

The Core Idea

A CRDT is a data structure with a merge function that is commutative, associative, and idempotent. Those three properties together mean you can apply updates in any order, any number of times, from any subset of nodes, and always converge to the same state.

• Commutative: merge(A, B) = merge(B, A) – order doesn’t matter
• Associative: merge(A, merge(B, C)) = merge(merge(A, B), C) – grouping doesn’t matter
• Idempotent: merge(A, A) = A – applying the same update twice is harmless

If your data structure satisfies these, you can replicate it across nodes with no coordination. Each node applies updates locally and gossips state to peers. Eventually everyone has the same state.

The Simplest CRDT: G-Counter

A grow-only counter. Each node maintains a vector of counts, one slot per node. To increment, a node adds to its own slot. To read the total, sum all slots. To merge two replicas, take the element-wise max.

Node A: [3, 1, 0]   Node B: [2, 2, 0]
Merge:  [3, 2, 0]   -- take max per slot
Total:  5

Node A can never decrement B’s slot and Node B can never decrement A’s slot. Merges always move counters up, never down. Convergence is guaranteed.

Adding Decrements: PN-Counter

A grow-only counter can’t go down. If you need decrements, use two G-Counters – one for increments (P), one for decrements (N). The value is P - N.

Each node increments its P slot to add and its N slot to subtract. Merges take the element-wise max of both vectors. The final value is the sum of all P slots minus the sum of all N slots.

Sets: The Interesting Case

Sets are where CRDTs get non-obvious. The naive approach breaks immediately: if Node A removes an element at the same time Node B adds it, what’s the right answer?

G-Set (grow-only set) sidesteps this: elements can only be added, never removed. Merge is union. Works perfectly but is obviously limited.

2P-Set (two-phase set) adds a remove set R alongside the add set A. An element is in the set if it’s in A but not R. Once removed, it can never be re-added. Merge is union on both sets.

LWW-Element-Set (last-write-wins) attaches a timestamp to each add and remove. On conflict, the higher timestamp wins. Simple, but you’re back to trusting clocks.

OR-Set (observed-remove set) is the one that gets this right. Each add operation generates a unique tag. Remove only removes the specific tags that were observed at remove time. If Node A adds element X (tag: t1) and Node B concurrently adds X again (tag: t2), then Node A removing X (removing tag t1) leaves Node B’s addition (tag t2) intact. The element stays in the set.

This matches user intuition: you removed the copy you knew about, not the one that was being added concurrently somewhere else.

State-based vs. Operation-based

There are two families of CRDTs.

State-based (CvRDTs): nodes gossip their full state. Merge is applied on receipt. The merge function must be a join on a join-semilattice – a partial order where every pair of elements has a least upper bound. Simple to reason about, but shipping full state can be expensive for large structures.

Operation-based (CmRDTs): nodes broadcast operations instead of state. No merge function needed on state – instead, the transport layer must guarantee exactly-once delivery and causal ordering. More efficient, but the delivery guarantee is a real constraint.

In practice, libraries like Automerge and Yjs use operation-based CRDTs with a hybrid approach: operations for real-time sync, periodic state snapshots for catch-up after disconnection.

Where They’re Used

Collaborative editing is the obvious one. Google Docs uses OT (operational transformation), an older approach with similar goals. Figma uses CRDTs. Notion, Linear, and most newer collaborative tools have moved toward CRDTs because the math is cleaner and they handle offline edits naturally.

Distributed databases: Riak’s eventual consistency model was built around CRDTs. Redis has CRDT support in Redis Enterprise for multi-region deployments. Azure Cosmos DB uses them internally for its multi-master replication.

Shopping carts: Amazon’s Dynamo paper mentions the shopping cart as a canonical use case. Adds and removes to a cart from multiple devices or regions converge without a coordinator.

The Tradeoff

CRDTs don’t work for everything. The merge semantics have to match what your application actually means.

A bank account is not a PN-Counter. If your account shows $500 and you withdraw $200 concurrently from two ATMs, merging the two PN-Counters would show $100 – which is correct math but incorrect banking. The constraint “balance must not go negative” requires coordination. You can’t avoid that with a CRDT.

The rule: CRDTs work when the merge semantics match application intent and when the application can tolerate the conflict-free resolution behavior. They’re not a universal replacement for coordination – they’re a way to avoid coordination in cases where the math lines up.

Vector clocks give you the diagnosis. CRDTs give you a structural guarantee that the diagnosis is never needed.

Google Spanner runs distributed transactions across a globally replicated database at scale. It can’t afford to block on coordinator failure. Its solution is to not use a standalone coordinator at all – it runs 2PC on top of Paxos groups, which means the coordinator itself is fault-tolerant.

The Core Move: Paxos Groups as Participants

In classic 2PC, each participant is a single node. It votes, acquires locks, and waits. If it crashes, the transaction is stuck.

Spanner’s participants are Paxos groups. Each shard of data is managed by a Paxos group – typically 5 replicas spread across datacenters. The Paxos leader for each group acts as the 2PC participant. Lock acquisition and voting happen on the leader, and the decision is durably replicated to the group before the leader responds.

If the coordinator crashes, a new coordinator can be elected (it’s also a Paxos group). If a participant leader crashes, a new leader in that group takes over – the lock state and vote are already replicated, so the protocol can continue.

The blocking problem is solved not by changing the protocol but by making every node in the protocol fault-tolerant.

TrueTime

Spanner adds something 2PC doesn’t have: globally synchronized timestamps, accurate enough to use for transaction ordering.

Google runs GPS receivers and atomic clocks in each datacenter and exposes a TrueTime API that returns a time interval [earliest, latest] rather than a single timestamp. The interval accounts for clock uncertainty. The guarantee is that the real current time lies somewhere in that interval.

Spanner uses TrueTime to assign commit timestamps. The rule: a transaction’s commit timestamp must be after the latest bound of TrueTime at the moment the coordinator decides to commit. This means Spanner waits for a brief period – typically 1-7ms – before declaring a transaction committed, to ensure that any other transaction starting after this one will see a strictly later timestamp.

This is the “commit wait.” It’s intentional latency, paid to guarantee that commit timestamps accurately reflect real-world ordering.

External Consistency

The property this buys is external consistency: if transaction T1 commits before T2 starts (in real-world time), T2 will see T1’s writes. This is stronger than serializability – serializable transactions can be ordered in any way consistent with their conflicts, even if that reorders real-world causality. External consistency preserves real-world order.

In practical terms: if you commit a write and then tell a colleague “it’s committed, go read it,” their read will see your write. There’s no window where a causally later transaction misses a causally earlier committed write.

Most distributed databases don’t provide this. They provide serializability, which is strong, but allows committed writes to be temporarily invisible to transactions that started after the commit in real time. Spanner’s external consistency closes that gap.

What the 2PC Layer Actually Looks Like

A read-write transaction in Spanner:

1. Client reads data, buffers writes locally (no locks yet)
2. Client sends commit request to the coordinator Paxos group
3. Coordinator runs 2PC with each participant Paxos group:
   • Sends PREPARE to each participant leader
   • Each participant leader acquires locks, logs the prepare record to its Paxos group, replies YES
   • Coordinator picks a commit timestamp (TrueTime latest + epsilon), logs it to its group
   • Sends COMMIT with the timestamp to all participants
4. Coordinator waits out the commit wait interval
5. Participants apply the writes at the commit timestamp, release locks
6. Client sees commit confirmation

The 2PC messages are still there. The key difference is that every step that touches durable state goes through a Paxos replication round first. The coordinator can crash and be replaced. A participant leader can crash and its successor has the full prepare state. Nothing blocks permanently.

The Cost

The commit wait adds latency. For single-region transactions it’s small (sub-millisecond when clocks are well-synchronized). For multi-region transactions where the coordinator and participants are in different continents, round-trip latency dominates and the TrueTime uncertainty is larger – you’re looking at tens of milliseconds.

Spanner is designed for this tradeoff: strong global consistency in exchange for latency that scales with geographic distance. For financial systems, global inventory, or anything that requires real external consistency, that tradeoff is often correct. For low-latency local transactions, it’s usually not.

Classic 2PC is blocking because the coordinator is a single node. Spanner’s answer is not to fix 2PC – it’s to make every node in the 2PC protocol a replicated state machine.

Two-phase commit (2PC) solves this. It’s been in production databases for decades. It also has a flaw that makes it unusable for anything long-running, which is why most modern large-scale systems don’t use it directly.

The Protocol

There’s a coordinator and a set of participants. The coordinator drives the protocol.

Phase 1 – Prepare

The coordinator sends PREPARE to every participant. Each participant:

• Writes the transaction to a durable log
• Acquires the necessary locks
• Replies YES (ready to commit) or NO (something failed)

Phase 2 – Commit or Abort

• If all participants said YES: coordinator sends COMMIT to everyone
• If anyone said NO: coordinator sends ABORT

Participants execute the decision and release their locks.

Coordinator         Node A        Node B
    |--- PREPARE ------->|             |
    |--- PREPARE ---------|----------->|
    |<-- YES ------------|             |
    |<-- YES ------------|-------------|
    |--- COMMIT -------->|             |
    |--- COMMIT ----------|----------->|

Clean path works fine. Every node ends up in the same state. Locks are held only for the duration of the protocol.

The Fatal Flaw

Crash the coordinator after phase 1 but before phase 2.

Participants are now stuck. They voted YES and locked their resources. They don’t know whether the coordinator decided to commit or abort. They can’t safely proceed in either direction – committing might be wrong if another participant said NO, aborting might be wrong if the coordinator was about to send COMMIT.

They wait. The locks stay held. The system is in limbo until the coordinator recovers.

This is the blocking problem. 2PC is a blocking protocol under failure. A single coordinator crash can stall the entire transaction indefinitely.

The Attempted Fix: 3PC

Three-phase commit inserts a pre-commit phase between prepare and commit:

PREPARE → PRE-COMMIT → COMMIT

Pre-commit lets participants infer the coordinator’s intent even if it crashes: if you received a pre-commit, the coordinator had seen all YES votes and was going to commit. You can safely commit yourself if the coordinator disappears after that point.

3PC eliminates most blocking scenarios. It costs one extra round of messages and still has edge cases under network partitions. It’s not widely used in practice – the complexity tradeoff isn’t worth it for most systems.

What Systems Actually Do

Most large-scale systems avoid 2PC for anything user-facing. Short, internal database transactions – a single service committing to one database – are fine. Cross-service transactions that could run for seconds or involve user interaction are not.

2PC is worth understanding because it makes the tradeoff concrete: you can have distributed atomicity, but you’re paying for it with availability under coordinator failure. Everything else in distributed transactions is a different point on that curve.

motivation

I was recently discussing with my CTO about databases. There was one point where we were thinking about the CAP theorem. Then he randomly mentioned that MySQL is not CA. I was surprised because I always thought that MySQL is a CA database at its core.

Now I had to search and understand why MySQL is not CA.

CAP theorem

We have always been taught in Engineering that CAP is one of the most important theorems in distributed systems and databases. We just accepted it. Time to question and learn ┏ʕ •ᴥ•ʔ┛

Here is what it actually means.

Imagine a bank server, when you check your account balance it should not randomly change it based on the server you are hitting and it should be consistent across all nodes. But in a distributed system, network failures (partitions) can occur. The CAP theorem states that you can only guarantee 2 out of 3 properties:

• Consistency (C): All nodes see the same data at the same time
• Availability (A): Every request receives a response, even if it’s not the most recent data
• Partition Tolerance (P): The system continues to operate despite network partitions

Here are practical examples of each combination:

CA Systems (Consistent + Available, but not Partition Tolerant)

Example: Traditional Relational Databases (like PostgresSQL in single-node setups)

Imagine a single PostgresSQL database server in a bank. When you check your balance:

• Consistency: Every time you query your balance, you get the exact same amount ($100.00)
• Availability: The system always responds to your requests
• Partition Tolerance: If the network connection to the database fails, you can’t access your account

This works perfectly in non-distributed environments, but fails when you need to scale across multiple data centers.

CP Systems (Consistent + Partition Tolerant, but sacrifices Availability)

Example: Apache ZooKeeper or etcd

Imagine a distributed configuration management system like ZooKeeper:

• Consistency: All nodes agree on configuration values (like which server is the leader)
• Partition Tolerance: If some nodes lose network connectivity, the remaining nodes continue operating
• Availability: During a network partition, some requests might be blocked to maintain consistency

In a ZooKeeper cluster, if a majority of nodes become unreachable, the system stops accepting writes to prevent inconsistent states.

AP Systems (Available + Partition Tolerant, but sacrifices Consistency)

Example: Amazon DynamoDB or Cassandra

Imagine a social media platform’s like counter system:

• Availability: The system always responds to requests, even during network failures
• Partition Tolerance: The system continues operating across multiple data centers despite network issues
• Consistency: During partitions, you might see different like counts on different servers

You might see a post with 100 likes on one page refresh, then 102 likes on the next refresh, as the systems reconcile the differences asynchronously.

Why MySQL is not CA

While MySQL can achieve CA in single-node deployments, it cannot be truly CA in distributed scenarios. When you add MySQL replicas or clustering solutions like MySQL Group Replication:

• During network partitions, MySQL must choose between consistency (rejecting writes) or availability (allowing potentially conflicting writes)
• MySQL’s synchronous replication can become unavailable during partitions if strict consistency is enforced
• Traditional asynchronous replication trades consistency for availability (so it becomes AP, not CA). Even with semi-sync replication, MySQL typically prioritizes availability over strict consistency—clients may see stale reads.

It doesn’t enforce strong ACID guarantees across replicas the way PostgreSQL doe

Why PostgreSQL is considered CA

Postgres in standalone mode: Definitely CA (strong ACID, always responds if node is up).

In replication setups:

PostgreSQL replication (especially with synchronous replication) prioritizes consistency first. A commit can be acknowledged only after replicas confirm the write (if you configure synchronous replication).

This means if replicas aren’t reachable, Postgres will prefer not to serve stale data (sacrificing availability to preserve consistency).

So Postgres is effectively CA (or CP in distributed setups).

This is why distributed databases like CockroachDB or TiDB are often preferred for truly distributed, partition-tolerant systems.

ʕ ● ᴥ ●ʔ