Lesson 144
MapReduce & Distributed Data
Map · Shuffle · Reduce
1:00How MapReduce spreads work across a cluster — the map, shuffle, and reduce phases, the word-count example, data locality, fault tolerance, and the combiner optimization.
By the end, you can
- Explain why a single machine is insufficient for very large datasets and describe the data-parallelism solution.
- List the three MapReduce phases in order and describe what each one does.
- Trace the word-count example end to end: identify what pairs the mappers emit, what the shuffle buckets contain, and what the reducers output.
- State the partition formula (`hash(key) mod R`) and explain why hashing the key matters.
- Explain data locality and why moving compute to the data is more efficient than moving data to compute.
- Describe how MapReduce achieves fault tolerance and why pure functions are the enabling property.
- Define skew and explain how a combiner mitigates it, including the commutativity/associativity constraint.
- Distinguish HDFS from local file systems and state the default replication factor.
- Compare MapReduce and Spark and explain why Spark is faster for iterative workloads.
Up next in Networking, Distributed Systems & Databases
