How many users does an A/B test need, and what to do when you don't have them

How many users an A/B test needs is not a matter of opinion. It is a number you can calculate. At the industry standard of 80% power and a 5% significance threshold, the rule of thumb asks for 16σ²/δ² users in each variant. On a site that converts at 3%, detecting a 10% relative improvement, the change that lifts 3.0 to 3.3, takes roughly 52,000 visitors per arm and more than 100,000 in total. If your monthly traffic is well below that, the answer is not to run the test a bit longer. An underpowered test does not leave you uncertain. It points you in the wrong direction and hands you a statistical significance badge while doing it.

Work out the number yourself

The formula is simpler than it looks: n = 16σ²/δ². The 16 comes from (1.96 + 0.84)² × 2, where 1.96 corresponds to the 5% significance threshold, 0.84 to 80% power, and the factor of two to the fact that both arms carry their own noise. For a binary metric such as conversion, the variance is σ² = p × (1 − p). And δ is the absolute difference you want to catch.

For a site converting at 3%, here is what each arm needs:

• 20% relative lift (3.0 to 3.6): about 12,900
• 10% relative lift (3.0 to 3.3): about 51,700
• 5% relative lift (3.0 to 3.15): about 207,000
• 2% relative lift (3.0 to 3.06): about 1.29 million

The interesting part is not the list but the ratio between its rows. Sample size scales with the inverse square of δ, so halving the effect you want to detect quadruples the traffic you need. If you cannot grow traffic in the short term, that relationship is your only real lever, and it says: test bigger changes. The derivation and the worked examples in this section come from Kohavi, Deng and Vermeer's KDD 2022 paper, A/B Testing Intuition Busters.

A low-powered test is misleading, not merely imprecise

The common assumption is that a test with thin traffic simply gives you a less precise answer. That is not what happens. At low power, a meaningful share of the results that come back significant are pure noise, and your tool labels them winners.

The same paper runs the numbers. If 10% of your ideas are genuine improvements, which is roughly the reported success rate at Booking.com, Google Ads and Netflix, then even in properly powered tests at 80%, 22% of your significant results are false positives. Drop the power to 20% and that figure climbs to 52.9%. More than half of your so-called wins are nothing at all.

There is a worse failure mode. As Gelman and Carlin showed, once power falls below 10%, the probability of getting the sign of the effect backwards approaches 50%. You ship a change that hurts the metric, believing it helps.

A concrete case. In December 2021 a real test circulated under the headline "which design radically increased conversions 337%?" It had 82 visitors in control and 75 in treatment, a p-value of 0.009 and an observed power of 97%. Convincing, on the face of it. The pre-test power for detecting a 10% change was 3%, and the false positive risk for that significant result worked out to 63%. The paper's authors note that across tens of thousands of tests at Airbnb, Booking, Amazon and Microsoft, they have never seen a change move conversion anywhere near that much. The practical threshold: A/B tests are useful for effects of reasonable size when you have at least thousands of active users, preferably tens of thousands.

Five options when your traffic falls short

• Stop testing small tweaks. A button colour comparison will never reach a conclusion at your volume. Test changes to the whole flow, the kind where a 20% effect is plausible. At 12,900 users per arm, that is within reach for many mid-sized sites.
• Move the metric closer to the change. Instead of judging a checkout edit by total revenue, judge it by checkout step completion rate. Variance drops and the sample you need shrinks. Accept the trade-off up front: the near metric can improve while the business result stays flat.
• Separate the decision from the test. Changes you cannot measure should go out behind feature flags as a staged rollout. The goal there is not to measure a lift but to cap the damage: open it to 5% of users, watch the error rate and support volume, keep the kill switch within reach.
• Switch to qualitative research. Five to eight moderated sessions will not tell you whether A sells better than B. They will tell you why people stall on that form, within a week, and that is usually the question you actually have.
• Decide some things without data. Arguing for a month about a difference too small to measure costs more than the decision itself. Make it cheap, make it reversible, move on.

Checking results mid-test breaks the test

Looking at results every day, asking whether they have turned significant yet, and stopping at the first green light quietly inflates your false positive rate. In Evan Miller's well-known calculation, a test inspected after every observation carries a real error rate of 26.1%, not 5%. Peek ten times and the threshold you read as 1% is really 5%.

Two legitimate fixes exist. Either you fix the sample size and the end date in advance and refuse to look at interim results for decisions, or you use a method built for looking: sequential testing with always-valid p-values folds continuous monitoring into the statistics, and most experimentation tools now ship it as a setting you can turn on. It is not free. For the same confidence it asks for somewhat more sample.

Decide which metric counts before you start

A test needs one decision metric, and that metric must not be something you can pump in the short term. Bing's own example is the warning. Making search results less relevant increases both query volume and ad clicks in the short run. Two metrics go up, the product gets worse, and users leave a few months later.

Put a handful of guardrail metrics next to the decision metric. These are the numbers you do not expect to improve but refuse to let slide: page load time, error rate, refund requests, support tickets. Speed matters here because its cost is measurable. Bing found that every 100 millisecond difference in performance moved revenue by 0.6%. Add your Core Web Vitals measurements to the guardrail list. And if the team disagrees about what a metric even means, settle that on one source of record before the test runs. A definition argument held after the results land throws the whole experiment away.

Three checks before you trust a result

1. Sample ratio mismatch (SRM). You configured a 50/50 split and ended up with 821,588 users in one arm and 815,482 in the other. That is 50.2/49.8, which sounds close enough. At that volume, the probability of the gap happening by chance is less than one in 500,000. A KDD 2019 study at Microsoft found an SRM in roughly 6% of experiments. When there is an SRM you do not correct the result, you void it and go find the assignment bug.
2. An A/A test. Run your tool against itself by serving both arms the identical experience. A correctly configured system should report no significant difference about 95% of the time. Microsoft caught hundreds of invalid experiments and misapplied formulas with exactly this check.
3. Bot and outlier rules. More than half of the requests Bing receives come from bots. Amazon traced a skewed test to library accounts placing bulk book orders. The rule is straightforward: write your cleaning rules before the test and apply them identically to both arms. André's 2021 work showed that removing outliers within each variant separately can push false positive rates as high as 43%.

Plan duration in whole weeks

Weekday users and weekend users are different populations. So a test runs for at least one full week, and when extended, it is extended in whole weeks. A test that hits significance on Tuesday does not get stopped on Tuesday.

The second reason is the novelty effect. People poke at anything new, the number inflates for a few days, then settles. The reverse happens too: established users who have to relearn an interface get slower in week one and recover afterwards. The first week is a poor predictor of the steady state, and returning users can take a few weeks to habituate. For interface changes, two weeks is a sensible floor.

If the number surprises you, it is probably wrong

Experienced data scientists follow Twyman's law: any figure that looks interesting or different is usually wrong. A surprising result is not something to celebrate, it is something to replicate. If the second run points the same way, now you have something to discuss.

These ratios help calibrate expectations. Across Microsoft, one-third of tested ideas improve the metric they were designed to improve, one-third change nothing, and one-third do harm. At Bing the positive rate is 15%, Booking.com, Google Ads and Netflix report around 10%, and Airbnb search comes in at 8%. Strong teams are wrong about most of their ideas. The Harvard Business Review account of Bing's ad headline change is the flip side: in 2012 a simple idea about how ad headlines were displayed sat for six months because program managers rated it low priority. Tested, it raised revenue by 12%, worth more than $100 million a year in the United States alone.

Write this down before you start

A single page is enough. Write it before the test and leave it alone while the test runs:

• The hypothesis and exactly what is changing
• The decision metric and the guardrail metrics
• The smallest effect worth detecting, and why that size
• Sample per arm and the end date it implies
• Early stopping rules, for harm only
• Exclusion rules: bots, internal IPs, outlier thresholds
• Who decides on the result, and at what threshold

If writing that page reveals the required sample would take three months at your traffic, you have learned the test is not happening. That beats finding out three months later, holding a false positive you have already shipped.