📝 MapReduce
Description
< What is it? >
MapReduce is a distributed batch-processing model for applying the same computation to a very large collection of records. It divides work into two main functions:
- Map: transforms each input record into one or more key–value pairs
- Reduce: combines all values that share a key into a final result
Between them, the system shuffles and sorts the map output so that every value for the same key reaches the same reducer.
Input records → Map → Shuffle and sort by key → Reduce → Result
Key points
< How the pipeline works >
Consider counting words in many documents:
| Stage | Input or output |
|---|---|
| Input | "red blue red" |
| Map | (red, 1), (blue, 1), (red, 1) |
| Shuffle and sort | red → [1, 1], blue → [1] |
| Reduce | (red, 2), (blue, 1) |
Many workers can run the map stage at once. The shuffle stage groups equal keys across all workers, and reducers can process different keys in parallel.
< Why it is useful >
- Scales horizontally: work can be split across many machines
- Handles failures: a failed map or reduce task can be rerun on another worker
- Moves computation near data: distributed systems can prefer workers that already store an input partition
- Simplifies parallel programs: the framework coordinates scheduling, shuffling, retries, and output files
< When it fits >
MapReduce is a good fit for large, independent batch jobs such as log analysis, counting events, building search indexes, or aggregating daily metrics.
It is less suitable for low-latency requests, interactive queries, or iterative algorithms that repeatedly reuse the same data. The shuffle and disk-based stages can add substantial network and storage overhead.
< Common implementations >
Google introduced the model, and Apache Hadoop MapReduce made it widely available for data stored in the Hadoop Distributed File System (HDFS). Modern systems such as Apache Spark often replace it for iterative workloads because they can reuse intermediate data more efficiently.
Comparison
| Approach | Strong fit | Main trade-off |
|---|---|---|
| MapReduce | Large, fault-tolerant batch transformations | Multiple stages can require expensive shuffle and disk I/O |
| Apache Spark | Iterative analytics and multi-stage pipelines | Requires memory and more operational tuning |
| Single-machine Python | Small data and rapid local development | Cannot efficiently scale beyond one machine |
Related ideas
- Apache Hadoop provides the Hadoop MapReduce engine, HDFS, and YARN.
- Apache Spark provides another engine for distributed processing.
- Data Processing