4 min readGuide
Shopify landing page A/B testing before you pick a builder
By Kratik Agrawal
Published Oct 13, 2026 Updated Oct 13, 2026
A Shopify landing-page A/B test should answer a specific buying question under consistent traffic and measurement. Write the hypothesis, outcomes and traffic requirement before selecting the tool.
Use the design checks below to distinguish page production, random assignment and adaptive routing, then decide whether your campaign can support the comparison.
Quick answer
Declare the hypothesis, metric, allocation, detectable effect and stop rule. Check traffic needs and product behavior before buying test capacity.
Can conversion rise while order value falls?
Yes. More purchases can bring less revenue per order. Define purchase rate, order value and revenue metrics before choosing a winner. Keep allocation and denominators consistent, then inspect the outcomes together. Use the result to answer the experiment’s declared business question before expanding the campaign.
I've launched a couple different versions of these pages that we're testing, and it all kind of flat lines. AOV goes down when conversion goes up.
Ecommerce operator · 2026-09-30 · lightly edited
Outcome metrics and report interpretation appeared in at least 2 selected merchant buying calls.
Selected from 7 relevant merchant buying calls; wording checked against the transcript and lightly edited; original retained internally.
Define the buying question and outcome metrics
Record the hypothesis, primary metric, allocation unit, minimum detectable effect and stop rule. A useful hypothesis identifies what the shopper cannot decide today and what the new page changes. Keep the main offer constant unless the offer itself is the experiment.
Use purchase or revenue metrics when that is the business question. Clicks and add-to-cart events can help explain a result but should not quietly replace the declared outcome. Shopify behavior reports describe sessions and funnel behavior; validate your experiment’s identifiers against the reports you actually use.
Shopify’s documented default funnel shows four stages: all sessions, sessions with cart additions, sessions that reached checkout and sessions that completed checkout. We counted the documented stage list on October 13, 2026. Those denominators matter when an experiment vendor reports a different conversion rate. Shopify behavior report documentation.
Purchase rate, revenue per session and contribution margin answer different questions. Track order value when offers or discounts change. Shopify notes that purchased-session counts can differ from orders because one session can contain multiple purchases. Match your experiment’s denominator to the store report.
Plan the sample and traffic assignment
Illustrative calculation, not Kinect or customer results. If a control page converts 3% of sessions and you want to detect 3.6%, a common two-proportion approximation with 95% confidence, 80% power and equal allocation requires roughly 13,900 sessions per version. That is about 27,800 total sessions, before exclusions or tracking loss. The approximation is n = 2 × 3.3% × 96.7% × (1.96 + 0.84)² ÷ (0.6%)².
A smaller page program may need a larger detectable change or a longer run. Do not lower the threshold after seeing results. Use a statistical calculator with your actual baseline and experiment assumptions before launch.
Ask whether the tool randomizes visitors, adapts routing or splits campaign URLs. Those methods support different comparisons. Unbounce presents A/B testing and Smart Traffic separately. Confirm which system produced the report; different ads or audiences can confound a page comparison.
Confirm whether assignment follows a session or visitor. Ask how repeat visits, cross-device use and unavailable cookies affect it. Define employee, test and bot exclusions before launch. Preserve raw counts and investigate inconsistent assignment before interpreting results.
Check the purchase path and decide how to read the result
Ask whether allocation happens per visitor or session, whether repeat visits keep their assignment, and whether traffic parameters survive. Inspect variant selection, discount handling, selling plans and the cart path on both versions. Validate the measurement independently of the visual editor.
Do not combine an audience change, a new ad, a new offer and a redesigned page and then attribute the result to layout. If several things must change together, describe the result as a campaign comparison and preserve that limitation.
Declare required traffic, duration, the deciding metric and technical stop conditions. Record early operational stops separately from favorable interim results. Report assignment, eligible traffic, conversions, revenue definitions and concurrent changes. An inconclusive test does not establish that pages perform identically.
Request traffic and outcomes by version, including repeat purchases, returns, cancellations and late conversions. Record baseline and denominator. Document mid-test changes and their effect on interpretation. Choose the next test around an unresolved purchase question; one campaign’s result does not establish a rule for every audience.
The experiment questions buyers raised
These selected calls raised outcome interpretation, enough time and traffic, and the experience actually being compared. Use them to pressure-test the experiment brief.
| Observed requirement | Minimum observed calls | Evaluation check |
|---|---|---|
| Outcome metrics and report interpretation | 2 | Define purchase, order value and revenue metrics, then identify the reports used to interpret them. |
| Enough time and traffic for a useful evaluation | 2 | Calculate traffic needs and evaluation duration before agreeing to a trial. |
| Which experience and traffic are actually compared | 3 | Specify the page versions, allocation and holdout rather than treating all optimization as the same test. |
Buy testing capacity for a test you can run
Proceed when the purchase path works, allocation is understood and the sample can answer the declared question. Otherwise finish the implementation checks or gather the required traffic first. Report an inconclusive result without inventing a winner.
Compare Shopify builders, Instapage alternatives and Replo’s published allowances against that brief. Use the page creation workflow, template guide and conversion-rate guide for their respective tasks. Review Kinect’s measurement method or discuss dynamic pages in a demo.
Frequently asked questions
How much traffic does a Shopify landing page A/B test need?
It depends on the baseline, detectable difference and design. The worked 3% to 3.6% example requires roughly 13,900 sessions per version under the stated assumptions; it is not a universal requirement.
Is adaptive routing the same as a randomized A/B test?
No. Routing adapts which experience a visitor receives; random assignment supports a different comparison. Confirm the method used in the report.
Which metric should decide the result?
Declare the business outcome before launch. Inspect purchase rate, order value and revenue with consistent denominators, plus the commercial guardrails your campaign needs.
What should we do with an inconclusive test?
Report the uncertainty and inspect sample size, implementation and traffic scope. The result does not establish that the pages perform identically.
