← All components
Analytics & processing
Batch processing
Large-scale offline jobs over data-lake files.
When to use
- Building search tries, recommendation features, reports
Options by provider
The first option is what diagrams show for that provider.
Amazon Web Services
- Amazon EMRManaged
- AWS GlueManaged
Google Cloud
- DataprocManaged
Microsoft Azure
- Azure DatabricksManaged
- Azure HDInsightManaged
Open source / self-hosted
- Apache SparkSelf-hosted
- Apache HadoopSelf-hosted
Used in
- Netflix
- Batch analytics — Aggregates viewing data to train models
- Payment gateway
- Reconciliation & settlement — Matches bank reports to the ledger; schedules payouts
- Search autocomplete
- Trie builder — Aggregates logs and builds new tries offline
- Google Maps
- Map data pipeline — Turns source map data into tiles, graph and index
- Driver booking & fleet management
- Payroll & payouts — Weekly earnings calculation