# A/B Test Design

**Framework:** Controlled experiment design  
**Use for:** Designing a valid A/B test before you run it, so the result actually means something  
**Time required:** 30–60 minutes

---

## How to use this template

1. Write the hypothesis BEFORE the test. "Let's see what happens" is not a test, it's a fishing trip.
2. Pick ONE primary metric. Multiple primary metrics is how you fool yourself into finding a "winner."
3. Calculate sample size and duration upfront, from your baseline rate and the smallest effect worth shipping. Underpowered tests waste traffic and produce noise.
4. Decide the stopping rule before you start. Peeking and stopping the moment it looks significant inflates false positives.
5. Remember significance is not importance. A statistically significant 0.1% lift may not be worth shipping.

---

## Header

| Field | Detail |
|-------|--------|
| Test name | [Name] |
| Owner | [Name] |
| Start / planned end | [Dates] |
| Surface | [Where the change lives] |

---

## Hypothesis

*State it as: because [insight], we believe [change] will cause [effect] on [metric].*

[Hypothesis]

---

## Metrics

| Metric type | Metric | Why |
|-------------|--------|-----|
| **Primary** (decides the test) | [One metric] | [What it measures] |
| Secondary (context) | [Metric] | [What it watches] |
| **Guardrail** (must not be harmed) | [Metric] | [e.g., load time, churn, revenue] |

*Exactly one primary. Guardrails catch a "win" that quietly breaks something else.*

---

## Power and sizing

| Input | Value |
|-------|-------|
| Baseline conversion rate | [%] |
| Minimum detectable effect (smallest lift worth shipping) | [%] |
| Significance level (α) | [0.05] |
| Statistical power | [0.80] |
| Required sample size per arm | [N] |
| Estimated duration to reach N | [Days / weeks] |

---

## Design

| Field | Detail |
|-------|--------|
| Control | [Current experience] |
| Variant(s) | [The change] |
| Traffic split | [50/50 or other] |
| Randomisation unit | [User / session / account] |
| Stopping rule | [Run to N, then read. No early peeking.] |

---

## Pre-registered decision

*Fill this in BEFORE launch. It removes the temptation to rationalise the result after the fact.*

- Primary improves, result is significant, no guardrail harmed → [Ship]
- Flat or not significant → [Don't ship / iterate]
- A guardrail is harmed → [Don't ship, regardless of the primary]

---

*Decide what counts as a win before you run the test, not after you see the numbers. More on significance, power, and correlation vs causation at biztechprimer.com.*