Most Shopify brands that look for an A/B testing tool are really looking for fewer lost sales. A testing app answers one question: did this change help? Before that come two others that decide whether the test is worth running. Where is the store losing shoppers, and what is that worth? And who builds the change, checks it and puts it live? This guide looks at the whole path, from friction to a tested fix, and at the questions that separate one tool from another.
From conversion leaks to tested fixes
Shopify store owners ask it in their own forums: “How do you know what to A/B test?” The answer starts before any test. A conversion leak is a place where shoppers who meant to buy stop: a size chart that sends them off the product page, delivery costs that appear only at checkout, an add-to-cart button out of reach on a phone. The evidence for it is in data the store already has:
- The funnel in Google Analytics 4: the step where the most sessions drop off, on mobile and desktop separately.
- The store’s own orders and sessions in Shopify: which pages and devices carry the revenue.
- Click maps, recordings and what customers say in surveys, support tickets and reviews.
A leak is worth fixing first when it is big, cheap to fix and safe for the brand. So each one needs a size: the sessions it touches, how many of them drop out, and the revenue those shoppers would have brought, as a range in $/month with the arithmetic behind it. Then the fix has to be built, reviewed, published to part of the traffic and measured against the original. Each handoff in that chain, from analyst to developer to whoever sets up the test, is a place where a good idea can stall.
In HI Engine the chain is one process. HI reads the store’s theme, orders, sessions and funnels from Shopify and Google Analytics 4, read-only until you ask for a change. It ranks each idea by impact, effort and brand risk, with an estimated $/month range and a confidence level. It writes the fix into the theme, and every fix runs as an A/B test against the original. Winners stay, losers roll back, and every result goes into the store’s knowledge base for the next idea.
Whichever tool you use, decide before a test starts what will count as a result: the metric (usually conversion rate, or revenue per visitor when the change could move order value), how long it runs (whole weeks) and the sample it needs. A test stopped at the first good-looking day proves nothing.
AI implementation and human oversight
Testing tools differ in how a change reaches the page. Some apps load a script that swaps content in the shopper’s browser after the page has loaded. That makes changes quick to set up, but the change lives outside the theme, and the original can flash before the variant appears. Other setups put the change into the theme itself, so it is served with the page like any other part of it. Which suits a store depends on its traffic, its theme and its team; check each tool’s own documentation, and Shopify’s developer documentation for apps and themes, for how a change is delivered.
AI tools that write theme code add a second question: who reviews it? A useful rule is that AI can do the work, but a person decides what goes live. Before approving a change, a reviewer should be able to:
- Preview it in the store itself, on a phone and on a desktop, not in a mockup.
- See exactly what changed in the theme, and that it passed Shopify’s own Theme Check.
- Read why the change was proposed, and what it is expected to move.
- Undo it in one step if anything looks wrong after it goes live.
This is how HI Engine works. Each change is written into the theme as native Liquid, JSON and CSS: no page builder, no injected scripts, no third-party layer on the storefront. Shopify Theme Check validates it, and it is published to a draft theme you preview in your own store. Nothing goes live without your approval; approved tests can be cancelled until they start, every change is logged, and rollback is one click.
Choosing a testing workflow
Some of the important features of each leading Shopify A/B testing app are easy to compare: price, setup time, the kinds of test it runs. The questions below go further, to the parts of the workflow where tests tend to stall. Ask them of any tool you look at, HI Engine included:
- Discovery: does it tell you what to test, from your own store’s data, or only run the tests you bring?
- Sizing: is each idea priced in money, with the reasoning shown, so you can rank it against other work?
- Implementation: who builds the change, where does it live (the theme or a script), and does it match your brand?
- Approval: can you preview every change in your store before it goes live, and roll it back in one step?
- Test design: are visitors assigned once and kept on one version, and is each change measured against a control?
- Reporting: are results judged on the metric set before the test, with mobile and desktop read separately?
- Where the team works: do updates reach you in the tools you already use, such as Slack, or only in another dashboard?
- The tools you have: does it read the testing and analytics tools you already run, or ask you to replace them?
In HI Engine, each store has its own Slack channel, where HI Agent posts every opportunity with its monthly estimate and a link to review it; you can also connect HI Engine to Claude. Every fix runs as an experiment, with visitors split 50/50 per browser, so no third-party A/B app is needed. If you already test in Intelligems, HI reads those results too.
To start, look at your own store. Enter your store URL on the free CRO ideas page: we walk your store on a phone, the homepage, a collection, a product page and the cart with a product in it, and you get three ideas, each next to a screenshot of what we saw, in about ten minutes. To go further, book a demo: a live walkthrough on your store’s data, so you know whether HI fits. If it does, the 14-day free trial starts from there. HI connects to Shopify and Google Analytics 4 and stays read-only until you ask for a change.
Start with a free conversion rate optimization audit
Read nextHow to A/B test on Shopify: a practical guide