What the calculator is actually checking
Running two versions of a headline, a subject line or a call to action against each other tells you which one performed better on the day. It doesn't tell you whether that difference would hold up if you ran the test again next week, with a different batch of visitors. That's the question a significance calculator answers: given the numbers you've collected, how likely is it that the gap between version A and version B is a real effect.
What it needs from you
The calculator can only work with what each version actually did, so it asks for the same two figures for each one.
- How many people saw that version (visitors or sessions)
- How many of them converted, clicked, or completed whatever action you're measuring
From those four numbers it works out a conversion rate for each version, then checks how far apart those rates are relative to how much variation you'd expect by chance alone, given the size of the groups you're comparing.
What a confidence level actually means
A confidence level of 95%, the setting most calculators default to, means that if there were truly no difference between the two versions, you'd expect to see a gap this large or larger by chance alone only 5% of the time. It's a statement about how surprising your data would look if the two versions were actually identical.
Why a small sample can point the wrong way
A test run on 40 visitors per version can easily show a large gap between two versions that is pure chance. Two more conversions on one version out of forty shifts that version's rate by 5 percentage points, and a calculator working from numbers that thin will often, correctly, say the result isn't significant, because there isn't enough evidence yet either way. The fix isn't to lower the confidence threshold until something looks significant. It's to keep collecting data until the sample is large enough for the calculator to have something real to work with.
Stopping a test early
Checking the result every day and stopping the moment it crosses the significance line raises the chance of a false positive, because you're really running many small tests and reporting the best one. Decide roughly how many visitors or how long you'll run the test before you start, and read the result once you reach that point rather than at the first flicker of significance.
What a significant result doesn't tell you
A significant result tells you that version B converted better than version A, in this test, on this traffic, for this action. It doesn't tell you why. The headline might be the reason, or the button colour, or the time of day each version happened to run, since anything that varied between the two groups could have played a part. Treat the result as evidence about which version to keep.
The calculator works from the numbers you type in, so a mistyped visitor count or a conversion event that fires inconsistently between the two versions will still produce a confident-looking answer, and that answer will be wrong. Check that both versions were tracked the same way before you act on the result.
Where this sits alongside the rest of your testing
The significance calculator answers a question the headline scorer and the CTA tester can't: not whether a version follows good structure, but whether a difference you've already seen between two live versions is real. Score a headline or a call to action before you test it, then run your live numbers through the A/B test significance calculator once you have enough of them to check whether the result is worth acting on.