Most A/B Tests Are Lying to You

Someone changes a button from blue to green. Three days later the dashboard says +23% conversion rate, and the change ships. Everyone is pleased.

The test proved nothing. The 23% was noise, the "significance" was manufactured by looking too often, and if you'd run it another week it would probably have gone the other way. A month later conversions are flat, and nobody connects the two facts — because the test said it won, and tests are supposed to be the grown-up alternative to opinion.

That's the real problem with bad A/B testing. It doesn't just fail to help. It launders a guess into a fact, and it does it while wearing the costume of rigour.

Here's what's actually going on, and the small number of rules that fix it.

The three ways a test lies

1. You didn't have enough traffic to detect anything

This is the big one, and it kills most tests before they start.

To detect a small effect you need a large sample. Not because statisticians are pedants — because a small effect is, by definition, hard to distinguish from random variation, and random variation is enormous when your numbers are small.

Rough sample sizes per variant, for the standard 95% confidence / 80% power setup:

Read that table honestly against your actual traffic.

If your landing page converts at 2% and gets 3,000 visitors a month, detecting a realistic 10% improvement needs roughly 78,000 visitors per variant — 156,000 total, which is over four years of your traffic. You cannot A/B test that page. Not "it'll be slow" — you cannot do it, and no amount of tooling changes the arithmetic.

This is the single most useful thing in this article, so I'll put it plainly: most sites do not have enough traffic to A/B test most changes. That isn't a failure of will or of tooling. It's a property of the numbers. Running the test anyway doesn't get you a smaller answer — it gets you a random one.

2. You looked at the results while it was running

This one is subtle, it feels completely harmless, and it is the reason so many tests "win" and then don't.

A p-value of 0.05 means: if there were no real effect, we'd see a result this extreme 5% of the time by chance. That guarantee holds for one look, at a pre-committed sample size.

Every time you check an in-flight test and consider stopping, you take another roll at that 5%. Check daily for two weeks and your real false-positive rate isn't 5% — it's somewhere around 20–30%. You will find a "significant" winner in a test between two identical pages if you look often enough and stop when it looks good. That's not a hypothetical; it's the guaranteed consequence of the procedure.

This is called peeking, and the ordinary workflow — open the dashboard each morning, ship it when it goes green — is a machine for manufacturing false positives. It is specifically the thing that makes an A/A test produce a winner.

The fix: decide the sample size before you start, and don't stop until you hit it. Write the number down. Then genuinely don't look, or if you must look, don't let yourself act. (Sequential testing methods exist that make continuous monitoring valid — but they're a different procedure with a different stopping rule, not permission to peek at a fixed-horizon test.)

3. You stopped as soon as it looked good

The evil twin of peeking. Results early in a test swing wildly — with 40 conversions in, a couple of lucky ones look like a 30% lift — and then converge toward the truth as the sample grows.

So a policy of "stop when significant" systematically stops during the swings. It doesn't just add noise; it is biased upward. You will consistently overestimate your wins, because you stop precisely when random variation is flattering you and keep running when it isn't.

This is why the effect sizes in the case studies are so much larger than anything you can reproduce. Nobody publishes the null results, and the wins were caught mid-swing.

What to do when you can't test

Given the table above, most of you can't run a valid test on most changes. That is fine, and it is much better to know it than to run an invalid one and believe the answer.

Test only the things worth 78,000 visitors. A rewritten pricing page, a fundamentally different offer, a new checkout flow — big swings on high-traffic pages where the effect might be large enough to detect. Not the button colour.

For everything else, use these instead — in rough order of reliability:

• Fix known defects. A form that fails on mobile, a CTA below the fold, a page that takes six seconds to load. These don't need a test, because there's no hypothesis: they're broken, verifiably. You don't A/B test whether users prefer a working form.
• Remove things. Deletions — a field, a step, a distraction — are far more reliably positive than additions. If you must ship an untested change, ship a removal.
• Watch behaviour instead of measuring outcomes. You need tens of thousands of sessions to detect a 10% lift, but you need about twenty session recordings and a scroll heatmap to see that nobody ever reaches your pricing table. Qualitative evidence has a vastly better signal-to-effort ratio at low traffic, and it tells you why, which a win rate never does.
• Test upstream, where the numbers are bigger. You may not have enough purchases to test the checkout, but you have plenty of pageviews to test a headline against scroll depth. Pick a metric far enough up the funnel that it actually moves.

If you are going to test, do it properly

Six rules. They're not negotiable, and together they take about ten minutes of setup.

1. Write the hypothesis down first, in falsifiable form.

"Moving the CTA above the fold on mobile will increase trial starts, because the scroll heatmap shows 60% of mobile visitors never reach it."

Not "let's try a green button." A hypothesis has a because in it, and the because should come from evidence you already have.

2. Compute the sample size before you start. Any online calculator will do it. Then compare it to your traffic and be prepared to abandon the test right there. This is the step everyone skips, and it's the one that would have saved them.

3. Pick one primary metric. One. If you track six and celebrate whichever moves, you're peeking across metrics instead of across time — same bug, different axis.

4. Run for whole weeks. Tuesday traffic is not Sunday traffic. A test that runs Monday to Thursday has measured the difference between your weekday and weekend audience as much as anything else. Always run in multiples of seven days, and never less than one full week even if you hit the sample size early.

5. Don't stop early. You wrote the number down. Honour it.

6. Run an A/A test first, once. Test the page against itself. If your tool reports a winner, your setup is broken — and you'll have learned that before you started making decisions with it, rather than after. It costs you one test slot and it is the cheapest insurance in the discipline.

The result you should expect

Most tests are inconclusive. That's the honest, unglamorous, professionally correct outcome, and it is information: it means the change you were sure about doesn't matter much, and you should stop arguing about it and go find something that does.

A field where most experiments succeed isn't a field with great ideas. It's a field with bad statistics.

The instinct to test everything usually comes from the same place as the instinct to stare at dashboards: it feels like rigour, it produces a number, and it defers the harder work of deciding. But a test you can't power is not evidence — it's a coin flip with a progress bar. Under the triage framework, the honest scoring of "A/B test the button colour" is low impact, high effort, low confidence, and it should lose to fixing the form that's throwing a 500.

What to do this week

1. Take your last "winning" test and check its sample size against the table above. If it's short, you didn't win — you rolled dice. That's uncomfortable and it's worth knowing.
2. Look up your real numbers: baseline conversion rate and monthly traffic on the page you want to test. Compute the sample size. Decide honestly whether testing is even available to you.
3. If it isn't — and for most pages it isn't — go and fix the known defects instead. They're higher confidence than any test you could run, and they don't need 78,000 visitors.
4. If it is, run an A/A test first, then follow the six rules. Set up the experiment properly, write down the stopping point, and then leave it alone.

The point of testing was never to generate wins. It was to stop you shipping things that don't work. A test that manufactures a winner from noise has inverted its own purpose — you'd have been better off with the guess, because at least you'd have known it was one.