Data Science Workflow & A/B Testing

Workflow · A/B tests · p-values · pitfalls

1:00

The six-stage data science loop and a complete A/B test — from framing a metric to interpreting p-values and avoiding the classic pitfalls.

By the end, you can

Recite the six stages of the data science pipeline in order and explain what each one does.
Convert a fuzzy goal into a precise OEC metric.
State the null and alternative hypotheses for a given A/B test.
Explain why random assignment is necessary for causal inference.
Calculate the required sample size direction given a change in MDE.
Interpret a p-value correctly and apply the p < alpha decision rule.
Distinguish Type I error, Type II error, and power, and state the recommended power target.
Identify the peeking, multiple-comparisons, Simpson's paradox, and significance-vs-importance pitfalls and explain why each inflates false conclusions.

Up next in Information Theory, Cryptography & Security