A store with a 2 percent conversion rate needs about 21,000 visitors per version, 42,000 in total, to reliably detect a 20 percent lift, and about 161,000 to detect a 10 percent lift. At 3,000 sessions a month that is 14 months and 54 months. If your traffic is that low, an A/B test cannot settle small changes: make big changes instead, judge them on orders rather than percentages, and do not stop a test early when one version looks ahead.
What “meaningful” means here
A conversion rate is orders divided by sessions, so with few orders it jumps around by chance. A test is meaningful when the gap between versions is bigger than that chance noise, at a stated risk: 5 percent chance of a false alarm and 80 percent chance of catching a real lift (common defaults; Evan Miller’s calculator lets you set both).
Visitors needed, worked out
Method: the standard sample-size formula for comparing two proportions with a two-sided z-test (normal approximation), 5 percent significance (z = 1.96) and 80 percent power (z = 0.8416), per version. The variance under the no-difference case uses the pooled rate p̄; under the real-lift case it uses each version’s own rate:
n = (1.96 x sqrt(2 x p̄ x (1 - p̄)) + 0.8416 x sqrt(p1 x (1 - p1) + p2 x (1 - p2)))^2 / (p2 - p1)^2
where p1 is your current conversion rate, p2 = p1 x (1 + lift), and p̄ is the average of the two.
Worked example, 2 percent conversion and a 20 percent relative lift (to 2.4 percent):
- p1 = 0.02, p2 = 0.024, p̄ = 0.022.
- First term: 1.96 x sqrt(2 x 0.022 x 0.978) = 1.96 x 0.2075 = 0.4067.
- Second term: 0.8416 x sqrt(0.02 x 0.98 + 0.024 x 0.976) = 0.8416 x 0.2074 = 0.1746.
- Sum = 0.5812; squared = 0.3378.
- Divide by (0.004)^2 = 0.000016: about 21,100 visitors per version (21,109 before rounding the steps), about 42,200 in total.
Results for other combinations (our arithmetic with the formula above):
| Baseline conversion | Lift to detect | Visitors per version | Total visitors | Orders in the test |
|---|---|---|---|---|
| 1% | 10% | 163,099 | 326,198 | about 3,260 |
| 1% | 20% | 42,694 | 85,388 | about 850 |
| 1% | 50% | 7,750 | 15,500 | about 155 |
| 2% | 10% | 80,684 | 161,368 | about 3,230 |
| 2% | 20% | 21,109 | 42,218 | about 840 |
| 2% | 50% | 3,826 | 7,652 | about 150 |
| 3% | 10% | 53,212 | 106,424 | about 3,190 |
| 3% | 20% | 13,914 | 27,828 | about 835 |
| 3% | 50% | 2,518 | 5,036 | about 150 |
The last column is the useful rule of thumb: whatever the baseline, detecting a 50 percent lift needs roughly 150 orders across both versions, a 20 percent lift roughly 840, and a 10 percent lift roughly 3,200. It is orders, not visits, that carry the signal.
How long that takes at your traffic
Using the 2 percent baseline rows and total visitors from the table:
| Monthly sessions | 50% lift (7,652) | 20% lift (42,218) | 10% lift (161,368) |
|---|---|---|---|
| 3,000 | 2.6 months | 14.1 months | 53.8 months |
| 10,000 | 0.8 months | 4.2 months | 16.1 months |
| 30,000 | 0.3 months | 1.4 months | 5.4 months |
Example line: 42,218 / 3,000 = 14.07 months.
What to do at each traffic level
| If your store gets… | Do this |
|---|---|
| Under 3,000 sessions a month | Do not A/B test small changes. Make a large change (offer, price, product page structure), wait for enough orders, compare before and after, and accept the answer is rough |
| 3,000 to 10,000 sessions | Test only changes you expect to move conversion by half or more; fix the sample size beforehand |
| Over 30,000 sessions | Tests for 20 percent lifts finish in weeks; test one thing at a time |
Two rules that matter as much as the sample size. First, decide the number in advance and do not stop when one version pulls ahead: Evan Miller’s write-up shows that checking repeatedly and stopping at the first significant reading raises false positives well above 5 percent (his example: stopping at the first 5 percent-significant reading, or after 150 observations, on a 50 percent baseline with no real effect gives a 26.1 percent false-positive rate, a worst case because the test is run after every observation). Second, traffic quality matters more than count; if the visitors are wrong for the product, no test result helps, and the traffic but no sales diagnostic comes first.
What to do with low traffic instead
- Fix the obvious before testing: the product page blueprint and trust signals guide.
- Use the conversion rate checklist and the funnel metrics guide to find which step loses the most people.
- Read what a store teardown looks for instead of testing blind.
Common questions
Can I trust a test that shows 95 percent significance after a week?
Only if you decided the sample size beforehand and the week reached it. Stopping the moment significance shows inflates false positives, per the source above.
Are these numbers exact?
They come from the standard normal-approximation formula at 5 percent significance and 80 percent power. Other tools use slightly different methods and can differ by a few percent; pick one and use the same numbers throughout a test.
Method: sample sizes are our arithmetic using the formula shown; the formula’s assumptions and the peeking warning come from the two Evan Miller pages linked above, checked 29 Sept 2026. No store data was used.