Ask
29

Every explanation of statistical significance defines it using itself, what does it actually mean?

I read a result described as statistically significant and I want to know what has been established.

Every explanation I find says something like: the result is unlikely to have occurred by chance. Then when I ask what unlikely means, the answer is that the p-value is below a threshold, and when I ask what the p-value is, the answer is the probability of the result occurring by chance. It closes into a loop and I come out knowing nothing.

I am not looking for the formula. I want to know what claim is being made, and - probably more importantly: what claim is not being made.

8 answers Share
Report

Answering anonymously, a moderator will review it first.

  • @assume_nothing_ana · 3w ago · 3 replies

    The loop closes because the usual phrasing leaves out the assumption that makes it meaningful. Put the assumption back and it stops being circular.

    The full statement is: if there were genuinely no effect, how surprising would data like mine be?

    So the procedure is:

    1. Assume there is no real difference. Not because you believe it, as a starting position.
    2. Ask what results that assumption would produce, given random variation and your sample size.
    3. Compare what you actually got against that.
    4. The p-value is how often that assumption would produce data at least as extreme as yours.

    A small p-value means: if nothing were going on, you would rarely see something like this. So either something is going on, or you got an unusual sample.

    Significant simply means that number fell below a threshold somebody chose in advance. The common threshold is a convention, not a law of nature.

    And note what the p-value is not, because this is the single most common error: it is not the probability that there is no effect. It is the probability of your data given no effect. Those are different quantities and swapping them is the mistake most of the confusing writing is built on.

    30
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @p_value_teacher · 3w ago · 2 replies

      Teach this and get it wrong most years. The version that works: assume there is no real effect, then ask how often data this extreme would show up anyway. Significant means the answer is rarely, under that assumption.

      The assumption is the part that goes missing in every casual explanation, and without it the sentence really is circular.

      22
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
      • @assume_nothing_ana · 2w ago

        Put the assumption back in and the loop opens. It is doing all the work and it is the part that gets dropped.

        13
        Share
        Reply

        Answering anonymously, a moderator will review it first.

        Report
  • @not_the_same_as · 3w ago · 2 replies

    The list of things it does not mean is more useful than the definition, and it is what you actually need to read a result.

    It does not mean the effect is large. With a big enough sample, a difference far too small to matter will be significant. This is the most common way significance misleads people.

    It does not mean the effect is real. It means it would be uncommon under the no-effect assumption. Run twenty tests on nothing and one will typically come out significant - that is what the threshold means, arithmetically.

    It does not mean the result will replicate.

    It does not mean the study was well designed. Significance says nothing about whether the right thing was measured, whether the groups were comparable, or whether the analysis was chosen after looking at the data.

    A non-significant result does not mean there is no effect. It frequently means the study was too small to detect one. Absence of evidence and evidence of absence are different, and reporting rarely distinguishes them.

    So when you see the word, the useful next questions are: how large is the effect, how big was the sample, how many things did they test, and was this the comparison they planned to make.

    26
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @p_value_teacher · 2w ago

      The list of what it does not mean is the practical half. In particular it says nothing about the size of the effect, and with a large enough sample a difference far too small to care about clears the bar comfortably.

      Which is why a significant result on its own does not tell you whether anything worth acting on happened.

      17
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
  • @effect_size_esra · 3w ago

    Read the effect size and its interval first. If the interval is wide, significance is not rescuing anything.

    10
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @p_value_teacher · 3w ago

    Unlikely to have occurred by chance, under what assumption. That missing clause is the whole problem.

    6
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @effect_size_esra · 3w ago

    The thing to look for instead, and the reason people who work with this stopped leading with significance: the effect size and the confidence interval.

    The effect size is how big the difference actually is, in units you can think about. Three minutes. Two percentage points. Half a mark. That is the number that tells you whether it matters.

    The confidence interval is the range of values consistent with the data. It carries the significance information: if the interval excludes no difference at all, the result is significant: and it also tells you how precise the estimate is, which significance alone hides completely.

    This is why an interval is more informative than a verdict. A result reported as significant might be an interval from a huge effect to a tiny one, and that is a completely different thing from a narrow interval around a modest effect, even though both get the same word.

    A practical reading habit: when you meet a significant result, look for the size. If the article does not give it, that is itself informative: a genuinely large effect is nearly always reported as a number, because it is the impressive part.

    And if you only remember one thing: significant is a statement about how surprising the data is, not about how big or how important the finding is.

    1
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report