p-hacking

P-hacking (Data Dredging)

Running many statistical tests and selectively reporting only those that produce P < 0.05 — exploits the fact that 5% of tests will show significance by chance even when nothing real is happening.

When you'd see it: In academic research (a major driver of the replication crisis), in data team work where analysts run many segment analyses and report only the 'interesting' ones, and in A/B testing when too many sub-group tests are run without correcting for multiple comparisons.

Why it matters: P-hacking produces false positives that look like real findings. When 20 tests are run on noise data, about 1 will show P < 0.05 by chance. The standard defenses: pre-registration (specify hypotheses before collecting data), Bonferroni correction (divide the threshold by the number of tests), or asking whether the hypothesis was specified before the analysis.

Common mistakes: Treating any P < 0.05 result as trustworthy without asking how many tests were run to find it. The more a dataset has been mined for significant results, the more cautiously each significant result should be interpreted.

Related P-value, statistical significance, null hypothesis, hypothesis testing