Articles · CRO & Growth
A/B split testing on Shopify
Published
Most ecommerce split tests never reach a result you could defend. That is worth knowing before you buy a testing tool, and it is not the argument against testing it sounds like.
The uncomfortable maths
The thing nobody selling testing software opens with: the sample size you need scales with how small an improvement you want to detect, and page changes usually produce small improvements. Detecting them reliably takes thousands of conversions per variant, not thousands of visitors.
For a store doing a few hundred orders a month, that means a single test running for months and only ever able to detect large changes. This is not a reason to avoid testing. It is a reason to know which category you are in before you build a programme around it.
What split testing is
Two versions of a page, shown to two halves of your traffic at the same time, measured against one outcome. The "split" is the traffic, not the page, which is the bit the name gets wrong. It is also written half a dozen ways, and anyone searching for what is AB split testing, or spelling it out as what is a b split testing, has usually met the term in a tool's marketing rather than in a statistics class. The mechanics are genuinely simple.
The difficulty is entirely in interpretation: knowing whether the gap between the two numbers is a real difference or the ordinary variation you would see if you split the traffic and changed nothing at all.
What to test, in order
Closest to the money, highest traffic, first. On almost every store that means the product page, then the cart, then collection pages. The homepage comes surprisingly far down: it gets attention because it is the page the team looks at most, not because it decides the most revenue.
Test substantial changes rather than button colours. A test needs a hypothesis worth being wrong about, and "does removing this step help" is worth running in a way that "is this green better" is not.
Four ways a test lies
- Stopping early
- You watch the dashboard, the variant pulls ahead, you call it. An early lead is what noise looks like before it settles, and a test stopped the moment it looks good is barely evidence at all. Decide the sample size first, then stop looking.
- Testing six things at once
- The variant wins, and you have no idea which of the six changes did it, or whether one of them was actively harmful and carried by the others. One substantial change per test, or accept that you are learning nothing transferable.
- Measuring the wrong outcome
- A change that lifts add-to-cart and reduces completed orders is a loss dressed as a win. Measure the outcome nearest the money, and check the step after the one you changed rather than only the one you were hoping to move.
- Something else changed too
- A campaign launched, a competitor discounted, an app updated and started injecting its own variant. Tests assume everything except your change stayed equal, and on a live store that assumption quietly fails more often than anyone admits.
Shopify specifics
The statistics do not change by platform but the practicalities do. Split testing shopify stores runs into three things in particular: the checkout is largely locked on standard plans, so the page with the most concentrated intent is often off limits; theme changes vary from trivial to development work with no obvious pattern; and apps that inject their own personalisation can interfere with a running test without anybody noticing.
Before trusting any result on shopify split testing, check what else is manipulating the page. That is the check most guides skip and it invalidates more tests than bad statistics do.
If you are too small to test
Most stores are, and the honest response is not to fake rigour with underpowered tests. Use qualitative evidence instead: session recordings, the questions your customer service team answers most often, and the checks that need no statistics at all, like whether your busiest collection page works on a phone.
Nearly every store has problems obvious enough that testing them would be a waste of the traffic. Fix those first. When volume genuinely supports it, running tests properly is the next step, and it sits inside the wider conversion work rather than being a separate purchase. If you want the shape of that engagement, what a CRO agency actually does covers it.
Common questions
- What is a b split testing, in plain terms?
- Showing two versions of a page to two halves of your traffic at once, then measuring which produced more of the outcome you care about. The word "split" refers to the traffic, not the page. It sounds simple, and the difficulty is entirely in knowing when the difference you are looking at is real rather than noise.
- How much traffic do you need before an ab split test is worth running?
- More than most stores have, which is the honest answer nobody selling testing tools leads with. To detect the size of improvement a typical page change produces, you generally need thousands of conversions per variant, not thousands of visitors. A store doing a few hundred orders a month can run tests, but it will take months per test and can only reliably detect large changes.
- What should we test first on a Shopify store?
- Whatever sits closest to the money and gets the most traffic, which is almost always the product page or the cart, and almost never the homepage. Test one substantial change rather than several small ones: two versions that differ in six ways tell you that something worked without telling you what.
- How long should a split test run?
- Until it reaches the sample size you calculated before starting, and through at least one full weekly cycle so weekday and weekend behaviour are both represented. Stopping when it looks good is the single most common way a/b split test results turn out to be wrong, because an early lead is what random variation looks like before it settles.
- Is Shopify split testing different from testing on other platforms?
- The statistics are identical; the practicalities are not. Themes make some changes easy and others need development, the checkout is largely locked on standard plans so the highest-leverage page is often off limits, and apps that inject their own variations can quietly interfere with a test. Check what else is manipulating the page before trusting a result.
- Should we use a testing tool or just change it?
- For a store below meaningful test volume, just change it, and use judgement plus qualitative evidence rather than pretending to statistical rigour you cannot reach. That is not a counsel of despair: most stores have obvious problems that do not need a test to justify fixing, and spending the tooling budget on finding those is better value.
Get in touch
Tell us about your store or your business and we'll reply personally. No wizard, no ten questions.
Stores we build and grow
