Data & storage

Sharding (partitioning)

Split one big dataset across many machines by a key so each holds and serves only part of it.

When a single database can't handle the write volume or data size, the data is split into shards. Each shard is an independent database holding the rows whose shard key maps to it.

Choosing the shard key is the most important decision: queries that include the key hit one shard; queries that don't must fan out to all of them.

See it in action

Step through the scenarios on the diagram.

1/3The app asks for user 42's profile.

App: — Reads and writes users
Shard router: — Computes the shard from the key
Shard map: — Which key range lives on which shard
Shard 0: — hash(userId) mod 3 = 0
Shard 1: — hash(userId) mod 3 = 1
Shard 2: — hash(userId) mod 3 = 2

How it works

Hash sharding: shard = hash(key) mod N (or a hash ring) — spreads load evenly.
Range sharding: contiguous key ranges per shard — good for range scans, risks hotspots.
Directory-based: a lookup table maps keys to shards, which allows rebalancing.
Typical keys: userId, conversationId, videoId, merchantId, city.

Trade-offs

Cross-shard joins and transactions become hard or impossible.
Hot keys (a celebrity, a viral video) can still overload one shard.
Resharding as data grows needs planning (virtual shards, consistent hashing).

Seen in

Components involved

Relational database Key-value / document DB Wide-column database

Related concepts

Consistent hashing Read replicas Horizontal scaling