Imagine a study examining whether a new exposure is associated with disease.
The researchers have:
• 8 exposure variables
• 6 health outcomes
• 5 predefined subgroups
They run statistical tests across these combinations.
Soon, there are dozens — or even hundreds — of possible comparisons.
Eventually, one association produces p = 0.03.
Is that evidence of a real finding?
The answer depends partly on what happened before that p-value appeared.
A conventional threshold of p < 0.05 is usually interpreted under a specific testing framework. When many hypotheses are tested, the probability of obtaining at least one apparently significant result by chance increases.
For example, if 20 independent null hypotheses are tested at α = 0.05, the probability of getting at least one statistically significant result by chance is:
1 − (0.95)²⁰ ≈ 64%
That calculation is simplified — real analyses often involve correlated outcomes and tests — but it illustrates the underlying problem.
The issue is not that the researchers conducted multiple analyses.
Multiple analyses are often necessary.
The issue is how those analyses are planned, selected and reported.
Suppose a paper reports:
> “Exposure X was significantly associated with outcome Y (p = 0.03).”
If X and Y were the primary exposure and outcome specified before analysis, that result has one context.
If researchers examined 8 exposures, 6 outcomes and several subgroups and reported only the combination with p = 0.03, the same number has a very different context.
This is why researchers distinguish between confirmatory and exploratory analyses.
Pre-specification helps readers understand which hypotheses were defined in advance.
When many hypotheses are part of the research question, methods for controlling the family-wise error rate or false discovery rate may also be appropriate, depending on the study.
But statistical correction does not replace transparent reporting.
Researchers should make it possible to understand:
• how many hypotheses were tested;
• which analyses were pre-specified;
• which analyses were exploratory;
• whether multiple comparisons were accounted for;
• and how the reported findings were selected.
There is a practical lesson here for both authors and readers.
A p-value does not exist in isolation.
Its interpretation depends on the analysis that produced it.
Sometimes the most informative question is not:
“Is p < 0.05?”
It is:
“How many opportunities did this study have to produce p < 0.05?”