Experimentation and A/B Testing

Programme overview

An experiment is the only reliable way to know whether a change worked. In this course you review Paystream's experiments in Python: a new signup flow, a homepage banner, a transfer-fee rise and a state-by-state rollout of cash-out agents. You'll see why 'users who did X' comparisons mislead, size a test before it starts, catch a broken randomiser with a sample ratio check, and simulate how peeking at results multiplies false alarms. Then you'll analyse conversion and skewed money metrics with confidence intervals and the bootstrap, read segments without fooling yourself, spot novelty, value a fee rise against the customers it costs, and estimate an effect without randomisation using difference-in-differences.

Curriculum

1. 01Why Experiment20 min

   • Why experimentWhy comparing users who did something with users who didn't can't tell you whether the thing worked, and how random assignment fixes that.
2. 02Designing a Test25 min

   • Designing a testWrite a hypothesis, choose one primary metric and the guardrails, set the smallest effect worth detecting, and calculate how many users and days the test needs before it starts.
3. 03Running a Test Properly15 min

   • Running a test properlyCheck that randomisation worked with a sample ratio mismatch test, and see by simulation why peeking at results and stopping early multiplies false alarms.
4. 04Analysing Conversion Tests25 min

   • Analysing conversion testsTest the difference between two conversion rates with a two-proportion z-test, report it with a confidence interval and relative lift, and say what it means in users and money.
5. 05Analysing Continuous Metrics15 min

   • Analysing continuous metricsCompare averages such as transactions or value per user with Welch's t-test, handle heavily skewed money metrics with a bootstrap, and decide whether a few big users are driving the result.
6. 06Segments, Multiple Tests and Novelty25 min

   • Segments, multiple tests and noveltyLook inside a result by segment without fooling yourself, correct for running many tests at once, and spot novelty effects that fade once users get used to a change.
7. 07Guardrails and Decisions25 min

   • Guardrails and decisionsWeigh a winning primary metric against guardrails that get worse, value both sides in money over a sensible horizon, and make a decision you can defend.
8. 08When You Can't Randomise25 min

   • When you can't randomiseEstimate an effect when a change was rolled out to some places and not others, with difference-in-differences, and check the parallel-trends assumption it depends on.
9. 09Final Project20 min

   • Final project: Paystream's experiment reviewPlan your final project, a full review of Paystream's four experiments and a design for the next one, and start by checking every test before trusting any result.

Assessment and certificate

You are assessed with a final assessment (pass mark 60%) and a final project. Your certificate is awarded when you meet those requirements. Paying for the programme does not award it on its own.