Grow & Convert
A/B Testing
Powered tests, honestly read.
(opens in a new tab) Most ecommerce A/B testing is underpowered and stopped early, which produces confident numbers that do not survive. The discipline is the product.
The stores we build, run and test, our own first.
Certified, not badge-only We run ecommerce A/B testing for Shopify brands across the UK and the US.
Read the Powerlete case studyEvery test we design for a client is the kind we have already run on our own two stores, with our own money on it. Most tests do not win, and a flat result written up properly is worth more than a winner nobody can repeat.
Conversion rate optimisationA/B testing agency
How we run a test whose result can be believed
Four steps, and the first one is arithmetic. Most stores cannot reach significance on most tests, so we work out what your traffic can actually answer before designing anything, and say plainly when the honest answer is not to test it.
-
Prioritise
A backlog from analytics, session recordings and the audit, ordered by expected value rather than by opinion.
-
Design the test
Hypothesis, single variable, success metric and required sample size agreed before build.
-
Run
Run to the calculated duration. No peeking, no early calls.
-
Read and bank
Result documented win, lose or flat, and the learning carried into the next test.
Split testing, done properly
Why most ecommerce A/B tests prove nothing either way
Most are stopped too early. A variant ahead after a few days is usually noise, and a test called at that point is a coin flip with a chart.
The rest fail more quietly: five changes bundled into one variant, so nobody knows which part won; a result read on add-to-cart while completed orders went the other way; and the losing tests left out of the report, which is how a programme ends up testing the same idea twice.
- Sample size calculated before a test starts, not after
- One change per test, so a result means something
- Read on revenue per visitor, not on click-through
- Losing and flat tests documented, because they are most of them
Ecommerce A/B testing
What is ecommerce A/B testing?
Ecommerce A/B testing shows two versions of a page to two groups of real visitors at the same time, and measures which one produced more revenue. The current page is the control and the change is the variant. Because both run side by side, the season and the ads affect them equally, which is what lets the difference be credited to the change.
Running one is the easy part. The hard part is knowing how many visitors a test needs before its result means anything, and holding to that number when the chart looks exciting on day three.
A/B testing services
What our ecommerce A/B testing service includes
Four rules, two on each side of the test. Break any one of them and the result stops being evidence.
Before a test starts
Powered properly
Sample size and duration calculated up front from your actual traffic and conversion rate. A test stopped the moment it looks good is a coin flip with a chart.
One change at a time
Bundling five changes tells you the bundle won and nothing about which part did. It also makes the loser impossible to unpick.
When it ends
Measured on revenue
Revenue per visitor, not clicks. A button that lifts add-to-cart and drops completed orders has cost you money while looking like a win.
The losers count
Most tests do not win. Written up anyway, because the pattern across them is where the real learning is, and hiding them is how a programme repeats itself.
Design
The hypothesis, the one change, the metric and the sample size, agreed before anything is built.
Ecommerce A/B testing
Our own checkout is where a test gets to be wrong first
Powerlete is our own sportswear brand. When we test a checkout change, a new section or a paid-social angle, we test it there first, on our own money, before it goes anywhere near a client build.
A losing test on Powerlete costs us the orders rather than a client, which is the cheapest place to find out an idea was wrong.
Our A/B testing stack
The stack we test on
A test is only as good as the traffic split and the number behind it, so this stack is built to be argued with. We break our own stores with it first, which is the cheapest place to be wrong.
Open on every test
Intelligems
Running the split, and keeping the traffic split honest
GA4
Sessions and revenue by landing page
Why a variant won or lost
Microsoft Clarity
Session recordings and rage clicks
Hotjar
Heatmaps and on-page surveys where recordings are not enough
heatmap.com
Revenue attributed to what people actually scrolled past
Whether the number is real
Upstack Data
Server-side tracking, so the numbers survive consent
Triple Whale
Blended attribution across paid and organic
Cloudflare Analytics
Real-user Core Web Vitals, on real devices
Run
Each test runs to the duration calculated before it started, and nobody calls it early.
Nour Hammour
Build, CRO and ongoing support
A/B testing agency
On a considered purchase, the test is rarely the button
Nour Hammour makes leather and tailored outerwear in Paris. We built the store, and we run conversion testing on top of it on a retainer.
At that price, the question a test asks is whether the page has answered the thing the customer is actually worried about, before anyone asks them to buy.
WUKA
Store build, then conversion testing
Shopify A/B testing
Testing what the page says, and in what order
WUKA makes leak-proof reusable period underwear, a product that has to be explained before it can be sold. We built the store, then ran a round of conversion testing on it.
On a product with this much to explain, a test is usually about what the page says and in what order: absorbency, fit, how many pairs someone needs. Rarely the shape of a button.
Ecommerce split testing
A test called early, against a powered one
The same tool runs both. The difference is what gets decided before the test starts, and what gets counted when it ends.
| Question | Called early | Powered |
|---|---|---|
| When it stops | The day the chart looks good | At the sample size set before it started |
| What changes | Several things at once | One thing |
| What it is judged on | Clicks or add-to-cart | Revenue per visitor |
| A losing result | Quietly dropped | Written up and kept |
| What you are left with | A number that may not repeat | A result, and a reason for it |
Bank
Win, lose or flat, every result is written up, and the learning goes into the next test.
A/B testing for ecommerce
The tracking came first, so the results could be trusted
Bessette sells curated luxury fashion, and two founders run it. Alongside conversion work, merchandising and paid media, we built a tracking setup designed to find the changes with real, meaningful impact.
That order is the point. A result is only worth banking if the revenue behind it was counted properly, and paid spend was scaled up as the tracking underneath it improved.
Reviews on the Shopify Partner Directory
Merchants, in their own words.
-
From start to finish, they were incredibly easy to work with. They understood the brief perfectly and delivered to a very high standard. Communication was excellent throughout, with quick and clear responses to any questions I had.
Tacky And Upsetting Store build or redesign · April 2026
-
The things [the team] have put in place and recommended for us have paid immediate dividend. We wish we had come in contact with [them] sooner.
ARTMARTIAL Store build or redesign · April 2026
-
I can’t recommend [the team] enough, [they] helped us with everything from product design, supplier communication to website building and SEO. We were able to build a brand off the back of all [their] information and help.
Reveliér Studios Business strategy guidance · April 2026
How we work with you
Testing programme
A backlog, tests designed and run to their calculated duration, and every result written up: win, lose or flat. Ongoing, because the learning is in the pattern across tests rather than in any one of them.
MonthlyDesigned and read by us
We write the hypothesis, set the sample size and read the result. Your developers build the variants. The cheaper answer when you have that capacity in-house.
AdvisoryConversion audit first
Where there is no backlog yet: analytics, recordings and the store itself, turned into a list of what is worth testing, ordered by expected value. The list is yours either way.
ProjectIs this the right thing to buy from us?
Worth a conversation
- Enough orders that a test can finish in weeks, not months
- A test was called a winner and the revenue never moved
- You want the losing tests written up as well as the wins
- Someone, yours or ours, can build a variant cleanly
Probably not this, yet
- Traffic is thin, and the maths says a test would never finish
- You want five changes tested at once
- You need an answer by the end of the week
Keep reading
Everything around a testing programme
Around testing
Traffic worth testing
The brands behind the stores we build and run.
Partners
Tools we have already run on our own stores.
The apps around a Shopify store decide as much as the theme does, and choosing between thousands of them is most of the work. We are a Shopify Plus Partner and we integrate the tools that sit around the platform. Every one of them runs on our own two stores before it reaches a client, so the recommendation you get comes from having paid for the thing and lived with it. When a brand already runs a stack that works, we integrate with it rather than rebuilding it to suit us.
How we use them
Straight answers
What if our traffic is too low to test?
You hear that before anything is built. The sample size comes from your own traffic and conversion rate, and where a test would take months to finish we say so, then point the effort at fixes the evidence already supports rather than pretending a test will settle them.
Why read a test on revenue per visitor?
Because it is the number that pays. A change can lift clicks or add-to-cart and still lose completed orders, or win orders at a lower value. Revenue per visitor catches both.
Do you ever stop a test early?
Not to call a winner. A test runs to the duration calculated before it started. If one is broken, a tracking fault or a variant that does not render, it is stopped and rerun rather than read.
What happens to the tests that lose?
They are written up like the winners: what changed, what we expected and what happened. Most tests do not win, and the pattern across the losers is where a programme learns what your customers care about.
Which testing tool do you use?
Intelligems for the split on most stores, with GA4 beside it. The tool matters less than the discipline around it: a sample size set in advance, one change, and a revenue read at the end.
Do we see a test before it goes live?
Yes. The hypothesis, the single change, the metric and the sample size are agreed with you before anything is built.
Is A/B testing the same as CRO?
It is one method inside it. Conversion rate optimisation finds where a store loses people; a test proves whether a fix works before it ships. If the diagnosis already exists, the testing can be bought on its own.
Tell us what you want to test
Send the store, roughly how many orders it takes a month, and the change you want proved. We reply personally, and if the maths says a test would never finish, that is the reply you get.
Stores we build and grow
Start a conversation
Talk to the people who would design the test.
Send us the store and the numbers you have. We will come back with what your traffic can answer, what we would test first, and whether a test is the right tool at all.
UK and US brands buy this from the same senior team, and the American day is covered from the start of it rather than the end. How we work with US brands.
Get in touch