statistical significance
Statistical Significance
A result is statistically significant when it is unlikely to have occurred by random chance alone — typically defined as P < 0.05, meaning less than a 5% probability of seeing this result if there were truly no effect.
When you'd see it: A/B test readouts, academic research, data team experiment summaries. Often expressed as 'significant' (shorthand) vs 'not significant' or NS. 'The result was statistically significant at P = 0.03' appears frequently in product experiment writeups.
Why it matters: Statistical significance is a filter for noise — it tells you the result is unlikely to be random. But it's a minimal bar: a result can be statistically significant and practically useless (tiny effect size), or statistically non-significant but real (underpowered study). Significance alone doesn't tell you what to do.
Common mistakes: Treating statistical significance as the same as practical significance. Statistical significance is determined by sample size as much as by effect size — large datasets produce significant results for trivially small effects that aren't worth acting on.
Study this in BizTech Primer →