← All concepts
Data & storage
Search indexing
Maintain an inverted index so text queries find matching documents in milliseconds.
Databases are poor at 'find products matching wireless headphones under $100'. A search engine tokenises text into an inverted index (term → documents), supports filters and facets, and ranks results.
See it in action
Step through the scenarios on the diagram.
Laying out diagram…
1/4A new product is saved.
- Catalogue service
- — Writes products
- Database
- — Source of truth
- Change stream
- — Every insert/update/delete
- Indexer
- — Builds search documents
- Search index
- — Inverted index + facets
- Search service
- — Parses and ranks queries
- Shopper
- — Types 'wireless headphones'
How it works
- Feed the index from change events (CDC) so it stays close to the source of truth.
- Shard the index and add replicas for query throughput.
- Autocomplete uses prefix indexes or tries.
Trade-offs
- The index lags slightly behind the database.
- Reindexing large catalogues takes time; use aliases to swap indexes.
Seen in
- YouTubeGlobal video upload, transcode, and streaming
- Twitter / XShort posts, follow graph and real-time home timelines
- AmazonHigh-traffic e-commerce marketplace
- Google MapsMap tiles, place search, routing and live traffic
- BookMyShowMovie and event tickets with live seat selection
- MakeMyTripFlights and hotels aggregated from many suppliers
- AirbnbSearch, book and message for vacation rentals
- NoBrokerBroker-free property rentals and sales
- Zomato / food deliveryRestaurant discovery, ordering and live delivery tracking
- Practo / teleconsultationFind doctors, book appointments, consult over video
- QuoraQuestions, answers, votes and a personalised feed
- LinkedInProfiles, connections, feed, messaging and jobs