A client writes to a Dynamo-style store — Cassandra, Riak, Scylla. The key's data lives on three replicas (RF=3), the write wants acknowledgment from two (W=2)… and replica C is down. What happens next is one of the most elegant small patterns in distributed systems:
A real techdiagrams document — download the JSON and import it into the editor.
The play, step by step
- The client sends the write to a coordinator (in these systems, any node can play this role for any request).
- The coordinator forwards to the three replicas responsible for key
k. A acks. B acks. C is dead. - W=2 is satisfied — the client gets a success. No retry, no error, no waiting on a timeout for C.
- Here's the pattern: instead of dropping C's copy on the floor, the coordinator writes a hint on a healthy node (D): "this write belongs to C — deliver it when C comes back."
- When C rejoins, D replays its stored hints, and C catches up on everything it slept through.
The name says it all: the write is handed off with a hint about where it truly belongs. Node D isn't becoming a replica of k — it's a mailbox holding C's undelivered mail.
Why this is clever
Availability without lying about durability. The success returned to the client is backed by two real, durable replica writes — the quorum was honestly met. The hint is extra: it accelerates C's recovery from "hours of anti-entropy repair" to "replay a queue on rejoin." Without hints, a rebooted C serves stale data until a full repair (see our Merkle trees post — the two patterns are teammates: hints catch the short outages cheaply, Merkle repair guarantees the long tail).
Write latency stays flat during failures. The coordinator never blocks on the dead node. The whole point of choosing a Dynamo-style store is that a machine dying is a Tuesday, not an incident — hinted handoff is a big part of what makes that true.
The trap: hints are not replicas
Now the part that bites people. It's tempting to think "three copies exist — A, B, and the hint on D — so we're fine." Two ways that's wrong:
- Hints don't serve reads. A read hitting C right after it rejoins (before replay finishes) can see old data. Your consistency still comes only from the quorum arithmetic: R + W > RF. W=2, R=1 with RF=3? A read served by just-rejoined C alone can be stale, hints or no hints.
- Hints expire. Real systems cap the hint window (Cassandra: 3 hours by default). If C stays down longer, hints are discarded — beyond that, only full anti-entropy repair heals the divergence. A hint is a fast path, never the guarantee.
There's also a subtler operational one: during a sustained failure, hint queues grow on the healthy nodes — a dead replica quietly converts into disk-and-write amplification on its neighbors. Under a long partition plus heavy writes, hinted handoff itself becomes a load problem, which is why the window is capped in the first place.
Break it on purpose
This pattern is the perfect chaos-experiment shape: kill a node mid-load, watch writes keep succeeding, bring it back, and check what reads saw in between. Import the diagram into techdiagrams.net, declare the guarantee you think you have ("a successful write is never lost", "reads see the newest acknowledged value"), and test whether your R/W/RF arithmetic actually holds it up. The gap between those two guarantees is exactly where production incidents live.