hypothesis testing
Hypothesis Testing
A formal procedure for using sample data to evaluate a claim about a population — sets up a null hypothesis, collects data, and calculates how likely the data would be if the null were true.
When you'd see it: Any designed experiment or A/B test. Hypothesis testing is the framework behind 'did our change work?' — you specify what you're measuring, how you'll know if there's an effect, and what significance threshold you'll use, before running the test.
Why it matters: The pre-specification requirement is what makes hypothesis testing trustworthy. You commit to the hypothesis before seeing the data, so the result isn't cherry-picked. Post-hoc analysis (looking at data and then generating hypotheses that fit) produces false positives at high rates.
Common mistakes: Running the test without pre-specifying the hypothesis and stopping criteria. Peeking at results mid-test and stopping early when you see significance inflates false positives significantly. Most A/B testing platforms now apply sequential testing corrections; be suspicious of tests that don't.
Study this in BizTech Primer →