p < 0.05 does not mean there is a 95% chance you are right

A study reports p = 0.03 and almost every reader takes away the same thing: there is a 97% chance the effect is real. Journalists write it that way, undergraduates are examined on definitions they can recite without believing, and a great many people who use statistics professionally would, under pressure, say something close to it.

It is wrong, and not by a technicality. The correct reading is a single sentence and it changes what the number is worth.

What a p-value actually is

The probability of observing data at least this extreme, if the null hypothesis were true.

Read the direction of that carefully, because the direction is the whole thing. It is a statement about the data, computed in a world where the null is assumed. It is:

P(data this extreme | the null is true)

and what almost everyone hears is:

P(the null is true | the data)

Those are different quantities, and conditional probabilities do not reverse. The probability the ground is wet given that it rained is very high; the probability it rained given that the ground is wet depends entirely on how many other things wet the ground round here. Same two facts, two different questions, two very different answers.

That is the whole error, and it has a name — the conditional gets transposed — but the name is less useful than the direction.

So what does 0.03 tell you

That if there were genuinely no effect, data this striking would turn up about three times in a hundred studies.

Notice everything that is not in that sentence. It says nothing about how likely it is that there is no effect, because you cannot get there without knowing how plausible the hypothesis was to begin with. It says nothing about how large the effect is. And it says nothing about whether a repeat study would find the same thing.

Why prior plausibility is not optional

Suppose you test a thousand hypotheses, of which a hundred are genuinely true. With a 0.05 threshold and reasonable statistical power, you might find perhaps 80 of the real effects — and you will also flag around 45 of the 900 nulls, because that is what a 5% false positive rate means.

You now hold roughly 125 significant results and about a third of them are wrong. Not because anyone made an error. That is the machine working as designed.

Push the proportion of true hypotheses down — an exploratory field, a large screen, a fishing expedition through a dataset — and the fraction of published “significant” findings that are false climbs above half. This is the core of Ioannidis’s much-cited 2005 argument that most published research findings are false, and it needs no misconduct at all.

Which is the practical upshot: the same p-value from a well-motivated hypothesis and from a hunch are not equally good evidence. The number is identical. What it is worth is not.

Four more things it is not

• Not the probability the result will replicate. Replication depends on the true effect size and the next study’s power, neither of which the p-value contains.
• Not a measure of importance. With a large enough sample, a difference too small to care about will produce an arbitrarily small p. Significance and magnitude are separate questions and only one of them is usually reported.
• Not 1 − probability of being right. p = 0.03 does not leave 97% anywhere.
• And “not significant” is not “no effect”. A p of 0.4 in an underpowered study is compatible with a large effect nobody had the sample size to detect.

Where 0.05 comes from

Nowhere principled. Fisher suggested it as a convenient conventional cut, and it became a threshold, and then in many fields a publication gate. Nothing in the mathematics distinguishes 0.049 from 0.051, but a career sometimes does — which is a decent short explanation of why replication has been such a problem.

It is also why the same dataset analysed with a few different reasonable choices — which covariates, which exclusions, which subgroup — will often produce at least one result under the line. Nobody has to cheat. Running twenty analyses and reporting the interesting one is enough, and it frequently isn’t even deliberate.

The American Statistical Association took the unusual step of issuing a formal statement on this in 2016, whose principles say plainly that p-values do not measure the probability that a hypothesis is true, and that conclusions should not be based on whether a p-value passes a threshold. When a professional body publishes a statement explaining what its own most-used tool does not mean, that is worth reading as a signal about how widespread the misreading is.

What to look at instead

Ask for the effect size and its confidence interval. That gives you the magnitude and the uncertainty in the units of the actual question, which is what you wanted to know in the first place — “somewhere between 2% and 19% better” is far more informative than “p = 0.03”, and it is honest about how little a small study pins down.

Then ask what the researchers expected before they looked, and whether they said so in advance. Prior plausibility is doing more work in your conclusion than any number in the paper, so it is better to state it than to leave it implicit.