← All components
Messaging
Event streaming
Durable, replayable, partitioned log that many consumers read independently.
When to use
- Event-driven microservices
- Analytics pipelines, CDC, activity feeds
Pitfalls
- Ordering is per partition only
- Consumer lag needs monitoring
Options by provider
The first option is what diagrams show for that provider.
Amazon Web Services
- Amazon MSKManaged
- Amazon Kinesis Data StreamsManaged
Google Cloud
- Pub/SubManaged
- Managed Service for Apache KafkaManaged
Microsoft Azure
- Azure Event HubsManaged
Open source / self-hosted
- Apache KafkaSelf-hosted
- RedpandaSelf-hosted
- Apache PulsarSelf-hosted
Related concepts
Used in
- YouTube
- Transcode jobs — Buffers new uploads so spikes only lengthen the queue
- Netflix
- Event pipeline (Keystone) — Carries play, pause and browse events to analytics
- Instagram
- Post & activity events — New posts, likes, comments and follows
- Twitter / X
- Tweet events — Every new tweet, like and follow
- WhatsApp / Messenger
- Message bus — Routes messages between chat servers
- Uber
- Trip & location events — Every ride status change and GPS ping
- Amazon
- Order events — Order placed, paid, shipped, cancelled
- Payment gateway
- Payment events — Every status change, in order per payment
- URL shortener
- Click events — One event per redirect
- Search autocomplete
- Query log — Every submitted search
- Dropbox
- Change events — One event per committed change
- Google Maps
- GPS probe stream — Anonymous location points
- WebSockets at scale
- Event bus — Durable, ordered stream of outgoing events
- BookMyShow
- Booking events — Confirmed, cancelled
- MakeMyTrip
- Booking events — Confirmed, failed, cancelled
- Airline reservations
- Ops & booking events — Delays, cancellations, bookings
- Airbnb
- Marketplace events — Listing changes, bookings
- NoBroker
- Listing events — Created, verified, updated
- Meeting scheduler
- Calendar events — Created, changed, cancelled
- Car rental
- Telematics stream — Vehicle updates
- Zomato / food delivery
- Order events — Every status change
- Urban Company
- Booking events — Confirmed, assigned, completed
- Porter / goods delivery
- Order events — Status changes
- Driver booking & fleet management
- Trip & telematics events — Trips, GPS history, vehicle health
- AstroTalk
- Session events — Started, minute charged, ended
- Quora
- Content events — New questions, answers, votes
- LinkedIn
- Activity events — Profile edits, posts, connections
- Buy now, pay later
- Credit events — Purchases, repayments, defaults
- Stock brokerage
- Market data stream — Normalised ticks
- Order & trade events — Status changes and fills
- Live sports scores
- Match event stream — Ordered events per match