Big Data & Data Engineering

By the end, you can

• Explain why "a lot of rows" is not what makes data "big," and what actually forces distributed systems.
• Name and define the 3 V's and give a concrete example of each.
• Trace data through the five stages of a standard pipeline (Sources → Ingest → Store → Transform → Serve).
• Explain why Spark is 10–100x faster than MapReduce on iterative jobs.
• Distinguish Spark transformations (lazy) from actions (trigger execution) and describe how lineage enables fault tolerance.
• Compare batch and stream processing on latency, throughput, and use-case fit.
• Describe how Kafka's append-only log, windows, and grace periods handle streaming data and late arrivals.
• Distinguish ETL from ELT and explain the advantage of loading raw data before transforming.
• Compare data lakes, warehouses, and lakehouses and describe the bronze/silver/gold pattern.
• Explain what Parquet, partitioning, and schema evolution each optimize in the storage layer.
• Describe the role of Airflow, data quality checks, and lineage in keeping pipelines reliable and trustworthy.