Ecommerce CRO testing: tools, traffic and when to skip it
What does CRO testing do?
An experimentation tool randomly splits visitors between two versions of a page, shows each group one version and records what each group does. That is an A/B test: the winner is judged on a metric you set in advance.
The point is a fair comparison. Without a split, you are comparing this month with last month. Last month had a different season, different traffic and different promotions.
The tool is the easy part. The design of the test is the hard part: what you change, what you measure and how many visitors you need before you decide.
Why do ecommerce brands use it?
Everyone in the room has an opinion about what makes a page convert. A test replaces the loudest opinion with a result. It also catches changes that feel right and do nothing, or quietly lower orders.
In our view, testing is one part of conversion rate work, not the start of it. Audits and research come first; tests check the ideas they produce. The same logic for ads, decided on margin and not on clicks, is in our creative testing framework.
What do you need to know before you test?
Three ideas decide whether a test can tell you anything:
- Sample size. How many users, conversions or regions a test needs to detect a real effect.
- Minimum detectable effect (MDE). The smallest lift your test design can reliably detect at the chosen power and error threshold.
- Statistical significance. A check on whether a measured difference is unlikely to be explained by chance alone, at a threshold chosen in advance. It does not say whether the effect is big enough to matter.
Together they give a simple rule. The smaller the change you hope to find, the more traffic you need to find it. A small shop testing a small change may wait a very long time for an answer.
A page test is not an ad test. A paper on Facebook lift studies notes that they differ from standard A/B tests, and that the experimentation literature did not describe how to calculate power and sample size for them1. For ad experiments, see how to read incrementality test results.
How do we implement it?
Check the traffic
Work out the minimum detectable effect first. If your traffic cannot detect it in a reasonable time, do not test yet.
Fix the known problems
Deal with what you already know is broken. Then test the ideas that audits and research produce.
Write the plan
Set the hypothesis, the metric and the stopping rule before the test starts. Judge it on orders and contribution margin.
Read every result
Let the test run until the plan says it is done. An inconclusive result is information, not failure.
When should you not use it?
What does it cost, in effort?
We do not give prices. In effort terms:
- Tool and set-up. Installing it, connecting it to your analytics and checking that the split works.
- Design. Writing the hypothesis, the metric and the stopping rule before the test starts.
- Build. Someone has to make the second version.
- Patience. The test runs until the plan says it is done, not until the chart looks good.
- Analysis. Reading the result, including an inconclusive one.
What are the pitfalls?
Stopping early
A winning chart on day three is a chart, not a result.
Measuring the wrong thing
Clicks are easy to move. Orders and margin are what pay the bills.
Calling significance importance
A result can be statistically clear and commercially trivial.
Treating "no result" as failure
An inconclusive test is information. It tells you the change is smaller than the test could detect.
Our take: Selective
This is our view, not a market fact. We rate ecommerce CRO testing and experimentation tools Selective.
Why:
- A test needs enough traffic to detect an effect in a reasonable time. Many shops do not have that for the pages they most want to change.
- When traffic is thin, other methods give better value: fixing known problems, learning from customers and checking the basics.
- We have no view on which tool is best, and we do not rank them.
Selective means: yes, where traffic and order volume can support a decision. Otherwise, fix the obvious things and measure them on orders and margin.
For ZFD: We judge a test on orders and contribution margin, not on click-through rate. Our CRO service covers audits and implementation, with no promised result. Test design for ads sits with our performance creative work.
FAQ
What is CRO testing?
It is running controlled experiments on a site, usually A/B tests, to see whether a change raises a chosen metric.
How much traffic do I need for an A/B test?
It depends on the effect you hope to detect. The smaller the effect, the larger the sample size. Work out the minimum detectable effect before you start.
Does statistical significance mean the change is worth making?
No. It says a difference is unlikely to be chance. It does not say the difference is big enough to matter.
Can I use A/B test formulas for ad tests?
Be careful. A paper on Facebook lift studies notes that they differ from standard A/B tests and need their own power and sample size calculations1.
Which testing tool should I buy?
We do not rank tools. Start with whether your traffic can support a test at all.
Let’s find what’s actually working.
Next step. Not sure your traffic can support a test? Talk to us about your setup.