A/B Test Design
Framework: Controlled experiment design
Use for: Designing a valid A/B test before you run it, so the result actually means something
Time required: 30–60 minutes
How to use this template
- Write the hypothesis BEFORE the test. "Let's see what happens" is not a test, it's a fishing trip.
- Pick ONE primary metric. Multiple primary metrics is how you fool yourself into finding a "winner."
- Calculate sample size and duration upfront, from your baseline rate and the smallest effect worth shipping. Underpowered tests waste traffic and produce noise.
- Decide the stopping rule before you start. Peeking and stopping the moment it looks significant inflates false positives.
- Remember significance is not importance. A statistically significant 0.1% lift may not be worth shipping.
Header
| Field | Detail |
|---|---|
| Test name | [Name] |
| Owner | [Name] |
| Start / planned end | [Dates] |
| Surface | [Where the change lives] |
Hypothesis
State it as: because [insight], we believe [change] will cause [effect] on [metric].
[Hypothesis]
Metrics
| Metric type | Metric | Why |
|---|---|---|
| Primary (decides the test) | [One metric] | [What it measures] |
| Secondary (context) | [Metric] | [What it watches] |
| Guardrail (must not be harmed) | [Metric] | [e.g., load time, churn, revenue] |
Exactly one primary. Guardrails catch a "win" that quietly breaks something else.
Power and sizing
| Input | Value |
|---|---|
| Baseline conversion rate | [%] |
| Minimum detectable effect (smallest lift worth shipping) | [%] |
| Significance level (α) | [0.05] |
| Statistical power | [0.80] |
| Required sample size per arm | [N] |
| Estimated duration to reach N | [Days / weeks] |
Design
| Field | Detail |
|---|---|
| Control | [Current experience] |
| Variant(s) | [The change] |
| Traffic split | [50/50 or other] |
| Randomisation unit | [User / session / account] |
| Stopping rule | [Run to N, then read. No early peeking.] |
Pre-registered decision
Fill this in BEFORE launch. It removes the temptation to rationalise the result after the fact.
- Primary improves, result is significant, no guardrail harmed → [Ship]
- Flat or not significant → [Don't ship / iterate]
- A guardrail is harmed → [Don't ship, regardless of the primary]
Decide what counts as a win before you run the test, not after you see the numbers. More on significance, power, and correlation vs causation at biztechprimer.com.