The forty-minute part
Writing the variants is the one step that genuinely got faster. Just not the way most people go about it.
The bad request: "write five variants of this landing page." What comes back is five rewordings. Same promise, same structure, different words. Test that and you learn nothing, because there is nothing to find out: there was no hypothesis.
The good request: every variant tests a separate claim. First you write down why you think the visitor is not buying — then you ask for the copy, one piece per claim.
Product: [one sentence]
Customer's own words: [3-5 quotes from reviews, support tickets, calls]
Current headline: [...]
Current conversion rate: [...]
Five hypotheses, five variants. For each one write down WHAT it claims:
V1 - does not get what this is -> the headline states the function, not the benefit
V2 - does not believe it -> a concrete number or reference goes under the headline
V3 - afraid of the risk -> the terms for backing out go next to the button
V4 - not urgent for them -> the opening paragraph states the cost of waiting
V5 - not aimed at them -> the headline names who it is for and who it is not
Rule: exactly ONE thing changes per variant against the control.
Use the customer's words, not marketing vocabulary.
That is the forty minutes. The quality of the input decides the result — the same logic that a working content brief runs on: the model cannot guess what you know about the customer.
What comes after is not forty minutes
This is where the fast part ends. Which variant is better is a statistical question, and there is an exact formula for it. The rule of thumb used in practice goes like this:
n ≈ 16 σ² / δ² per variant
σ² = the variance of the metric (for conversion, p(1-p))
δ = the difference you want to detect
The 16 is not a magic number: it falls out of two constants, the usual 5% significance level and 80% power. The part that matters is the denominator: δ is squared.
Here is what that means for a two-arm test, in total visitors (5% significance, 80% power, calculated with the exact formula):
| Current conversion rate | What you want to detect | Total visitors needed |
|---|---|---|
| 1% | +10% (1 → 1.1%) | 326,184 |
| 2% | +10% (2 → 2.2%) | 161,358 |
| 2% | +20% (2 → 2.4%) | 42,212 |
| 2% | +50% (2 → 3%) | 7,644 |
| 5% | +10% (5 → 5.5%) | 62,462 |
| 5% | +20% (5 → 6%) | 16,310 |
| 10% | +20% (10 → 12%) | 7,676 |
At 500 visitors a day, the case one row below the top — 2% conversion, a +10% effect — would run for 323 days. The same thing for a +20% effect is 84 days, for +50% it is 15.
The squaring rule
This is the one genuinely important sentence in the post: detecting an effect half the size takes four times as many visitors. In the table, at a 5% conversion rate, +20% is 16,310 visitors, +10% is 62,462, and +5% is 244,242. Not linear — squared.
That is why "I will just let it run longer" does not work. "Longer" here does not mean one and a half times the time, it means four times.
What size of effect can you expect at all
Most plans fail because somebody expects a +30% improvement from a headline swap. The experimentation literature is far more modest: more than half of all ideas produce no measurable improvement whatsoever, and in some fields the failure rate runs above 90%. The ones that do work typically move key metrics by 0.1–1.0% — at one large search engine's team roughly one experiment in five hundred clears the high-return threshold.
Read that together with the table — but watch out, these are two different numbers. The table shows a relative lift: "+10%" means 2% going to 2.2%. The 0.1–1.0% is the absolute improvement once diluted to overall impact. The shared lesson is the same: the effect you can realistically get is small, and detecting a small effect takes traffic in the hundreds of thousands — which is why, for most small-business landing pages, an A/B test is the wrong instrument. That is not a failure, just arithmetic.
So what do you do with low traffic
Four things work, and none of them is an A/B test.
- Test big things, not words. A 0.3 pp difference is undetectable; a completely different offer or a different audience can move the number enough to show up with fewer visitors. If you are going to test, test something where +30–50% is realistic.
- Switch to a less jumpy metric. The more variance a metric has, the more samples it takes — revenue per visitor jumps around far more than "opened the price list". Measuring the closer, more frequent event tells you something with a smaller sample; just be careful, that still does not guarantee it turns into a purchase.
- Ask five people. Five moderated click-throughs fit into a weekend, and they expose the comprehension failures in your copy more reliably than an underpowered test does.
- Use the customer's words, and do not test them. If thirty people write the same sentence in your reviews, that is not a hypothesis, that is data. Put it in the headline.
The cost of peeking
The most common mistake is not a bad variant, it is stopping early. If you check the p-value daily and declare a winner at the first significant moment, your type I error rate does not stay at 5%.
By one measurement, at as few as 10,000 samples the false positive rate can rise fivefold — that is, instead of the nominal 5%, roughly every fourth "winner" is actually nothing. This is the most common reason a winning variant fails to deliver once it goes live.
There are two ways out. Either you fix the sample size and the run time in advance and do not look until you get there. Or you use a sequential method that controls the error at every look — several testing platforms offer that built in. What you cannot do is watch a fixed-horizon test continuously.
How many variants fit
Every extra variant makes two things worse at once.
One: the chance of finding a winner by accident goes up. Counting the control, with four arms — the control plus three variants — it is 14.3%, with ten arms it is already 37% that at least one false positive turns up.
Two: if you correct for that — with a Bonferroni correction, for instance — the traffic you need jumps. At a 5% conversion rate, for a +20% effect: two arms need 16,310 visitors, six arms need 72,804. Four and a half times as many.
Generating five variants is therefore a writing method, not a test plan. Write five, pick the best two, and test those two.
How long should it run
At least a week, preferably two — and always whole weeks, so the weekend does not skew it.
The reason is not only sample size. There is a novelty effect: whatever is new, users react to it differently in the first few days. In one documented case, on the first day after a feature launch 20% of the feedback was about it, most of it negative; by the second week that was down to 4%, then 2% — while the metrics improved throughout. Anyone who decides after day one is measuring that something is new, not that it is good.
So does AI write better copy or not?
There is a good measurement of this, and the answer is sobering. Three pre-registered randomised experiments ran on a US wine retailer's real newsletters, 27,500 customers each, two weeks apiece: a human copywriter, a language model, and the two combined — only the body copy differed.
All three mailing arms doubled gross margin against the control group that got no email. Between the human and the AI arms, though, no difference could be detected, in any of the experiments. The AI still won, just not where you would expect: the model licence came to around a thousand dollars a year, three copywriters cost $375,000 a year — which is why the AI arms came out ahead on annual net profit.
So AI does not write better copy, it writes copy just as good at a fraction of the cost. That is the right expectation. If somebody promises you that generated copy lifts conversion by itself, ask for the measurement behind it.
For ad creative the picture is different, and there the advantage has been measured — producing ad creative at volume is the post that goes through that in detail.
Common misreadings
- "We have a winner, it is 0.5% better." At two hundred visitors that is noise. The table tells you when it stops being noise.
- "I will just run it longer if nothing comes out." Halving the effect quadruples the traffic you need. "Longer" usually means months.
- "Ten variants give me more chances." More chances at a false winner: at ten, 37%.
- "A test is objective, so it cannot be got wrong." Stopping early can multiply the false positive rate by five. A test is objective only if you fix its rules in advance.
- "AI copy converts better." By the most thorough measurement, exactly as well. Its advantage is the price, not the quality.
What to take away
Writing the variants takes forty minutes; deciding is a question of traffic. Work out before the test how many visitors it needs — and if you do not have them, do not run the test, decide from the customer's own words instead, and test bigger things. Generating got faster; proving did not.