Every explanation of statistical significance defines it using itself, what does it actually mean?
I read a result described as statistically significant and I want to know what has been established.
Every explanation I find says something like: the result is unlikely to have occurred by chance. Then when I ask what unlikely means, the answer is that the p-value is below a threshold, and when I ask what the p-value is, the answer is the probability of the result occurring by chance. It closes into a loop and I come out knowing nothing.
I am not looking for the formula. I want to know what claim is being made, and - probably more importantly: what claim is not being made.
@assume_nothing_ana · 3w ago · 3 replies
The loop closes because the usual phrasing leaves out the assumption that makes it meaningful. Put the assumption back and it stops being circular.
The full statement is: if there were genuinely no effect, how surprising would data like mine be?
So the procedure is:
A small p-value means: if nothing were going on, you would rarely see something like this. So either something is going on, or you got an unusual sample.
Significant simply means that number fell below a threshold somebody chose in advance. The common threshold is a convention, not a law of nature.
And note what the p-value is not, because this is the single most common error: it is not the probability that there is no effect. It is the probability of your data given no effect. Those are different quantities and swapping them is the mistake most of the confusing writing is built on.
Reply
Report
@p_value_teacher · 3w ago · 2 replies
Teach this and get it wrong most years. The version that works: assume there is no real effect, then ask how often data this extreme would show up anyway. Significant means the answer is rarely, under that assumption.
The assumption is the part that goes missing in every casual explanation, and without it the sentence really is circular.
Reply
Report
@assume_nothing_ana · 2w ago
Put the assumption back in and the loop opens. It is doing all the work and it is the part that gets dropped.
Reply
Report