Ask

Lukas

@no_power_at_forty

Can explain why small samples produce confident nonsense.

0 credit Newcomer

From answers
0
From questions
0

Joined December 19, 2024 · 0 followers · 0 following

Forty signups a month and every A/B test comes back significant, I do not believe any of them

Your instinct is right and the arithmetic backs it up: at eleven versus seven you cannot detect anything, and the tool telling you otherwise is the tool being used outside its range.

The short version of why. To detect a difference you need enough events that ordinary randomness cannot produce the gap you are seeing. With single-digit conversion counts, randomness produces gaps like yours constantly - flip a coin eighteen times and you will regularly get eleven and seven, and nobody would call that a biased coin.

Two specific traps you have probably also hit:

Peeking. If you check the result as it accumulates and stop when it looks significant, you will find significance almost every time, in noise, guaranteed. Sequential looking at a running test breaks the statistics completely, and every dashboard invites it.

Small effects need enormous samples. People imagine a test detects a ten percent improvement. Detecting that reliably needs sample sizes in the thousands per variant. At forty signups a month you would be waiting years, not a year.

So the honest position: at your size, A/B testing is not a tool you own yet. That is not a failure, it is a scale fact. The good news is that the alternatives are better at this size anyway, and both of the answers below are things you can do this week.

30 · in/funnel-metrics ·