Your sharding is perfect. Sixteen shards, consistent hashing, keys spread beautifully evenly. Then one post goes viral — and shard 3 is at 100% CPU while the other fifteen idle at 2%. Adding shards does nothing. Welcome to the hot-shard problem, the failure mode that perfect hashing cannot save you from:
A real techdiagrams document — download the JSON and import it into the editor via File → Import.
Why "just add shards" fails
A hash function distributes keys evenly — it says nothing about the traffic per key. Real workloads are zipfian: a handful of keys carry a huge share of all requests. The celebrity's post, the viral video, the flash-sale SKU, the one tenant who is 40% of your revenue. A perfect hash dutifully places that key on exactly one shard — and now 30% of your entire load has a single machine's ceiling. Re-hashing, re-sharding, doubling the fleet: the hot key still lands somewhere, whole.
There's a quieter cousin: the hot shard without a hot key — range-sharded data where the newest range takes all the writes (timestamp-prefixed keys, auto-increment IDs). Different cause, same melted machine, and worth ruling out first: is it one key, or one range?
The mitigation ladder
Work down this list; each rung is more invasive than the last.
0 · Detect it. You can't fix what you can't name. Per-shard CPU tells you which shard; only per-key metrics tell you which key. A top-K heavy-hitters sketch (count-min + a small heap) is cheap enough to run always-on — and turns "shard 3 is sad" into "post 8412 is why."
1 · Cache the hot key — kills read heat. The viral post is the same bytes for everyone; serving it from a near-cache (or the client) removes 99% of reads before they touch the ring. Two traps hide here: stampede — when the cache entry expires, ten thousand requests miss simultaneously and melt the shard you just saved (fix: request coalescing — one loader, everyone else waits — plus TTL jitter); and uselessness against writes — likes, counters and bids don't cache.
2 · Salt the key — kills write heat. Split the hot key into key#0 … key#15, each hashing to a different shard; writers pick a salt at random, readers fan in and merge. This is the classic sharded counter: sixteen partial counts, summed on read. You've traded one hot writer path for a slightly costlier read — almost always the right trade. The subtlety: salt only the hot keys (detection, again), because salting everything multiplies every read by N.
3 · Rebalance with a bound. Bounded-load consistent hashing caps any shard at c× the average load and spills the excess to the next node on the ring — it contains moderate skew automatically. For range shards, split the hot range and move half — this is exactly what DynamoDB's adaptive capacity and Bigtable's tablet splitting do for you. Know the ceiling: no rebalancer can split a single key. That's rungs 1 and 2's job.
4 · Isolate the whale. When one tenant or one entity is persistently enormous, stop treating them as ordinary data: give them a dedicated shard (or a dedicated cell), with their own quotas and their own blast radius. The heavy-hitter stops being an incident and becomes a line item. This is the pattern behind "celebrity infrastructure" at large social networks — the biggest accounts genuinely run on different plumbing.
The anti-patterns that manufacture hot shards
Worth naming, because they're all avoidable at design time: timestamp-prefixed keys (all writes hit the newest range — prefix with a hash instead), auto-increment partition keys (same disease), user_id as the partition key for a B2B product (your largest customer is a hot shard), and caching without stampede protection (rung 1 done badly is worse than not doing it).
See it melt, then fix it
The flash-sale template in our gallery composes the full defense — sharded counters, token buckets, cache warming — as an importable architecture. Or import this post's diagram, put real numbers on the edges, and watch what 95%-on-one-key does to a design before your traffic does it for you.