← All concepts
Reliability & correctness
Leader election & coordination
Agree on which node is in charge or who owns which shard, using a consistent coordination service.
Some jobs must run on exactly one node (schedulers, shard owners, key-range allocation). A consensus-backed store such as etcd or ZooKeeper grants leases; if the leader dies, its lease expires and another node takes over.
See it in action
Step through the scenarios on the diagram.
Laying out diagram…
1/4Worker 1 holds the leader lease and renews it every few seconds.
- Worker 1
- — Current leader
- Worker 2
- — Follower
- Worker 3
- — Follower
- Coordination store
- — Grants a lease to one worker
- Nightly job
- — Must run exactly once
How it works
- Leases with TTL and heartbeats.
- Watches notify nodes when configuration or ownership changes.
Trade-offs
- The coordination cluster itself must be small, reliable and fast.
Seen in
- URL shortenerShort links with billions of redirects
- API rate limiterFair-use limits checked on every request in under a millisecond
- Search autocompleteSuggestions on every keystroke in under 100 ms
- Web crawlerFetch billions of pages politely and without repeats
- Redis ClusterDistributed in-memory data store