Skip to main content
Use this tutorial when you need to turn one Analytics dashboard into a clear decision: ship the variant, keep running the Experiment, or revise the Experiment.

​
If you remember only four things

  • Pick exactly one primary metric before the Experiment starts.
  • Use guardrail metrics to catch side effects, not to create a second primary winner rule.
  • Check visitor count, run time, and traffic split before you decide.
  • Do not ship if guardrails show a business risk that matters.

​
Start by choosing one primary metric

Your primary metric is the one number that decides whether the Experiment worked. Choose it before the Experiment starts. For metric definitions, see Analytics metrics, Traffic and conversion metrics, and Revenue and profit metrics.

​
Choose guardrails after the primary metric

Guardrails are the checks that stop you from shipping a harmful variant. Pick them from the business goal and the risk you expect. Use two or three guardrails. They do not need to beat control. They need to stay healthy enough for the decision to make business sense.
  • If the goal is more purchases and the primary metric is Conversion Rate, guardrails may include Revenue per Visitor or Profit per Visitor. This checks that more purchases are not coming from lower-value or lower-margin orders.
  • If the goal is higher order value and the primary metric is Average Order Value, guardrails may include Conversion Rate and Profit per Visitor. A higher threshold or bundle can raise order value while reducing purchase likelihood or profit per visitor.

​
Check whether you have enough data to decide

Review the sample before you choose the next action.
Running for 1-2 weeks helps include weekday and weekend behavior. High-traffic stores may reach a useful visitor count sooner. Low-traffic stores often need more time.
There is no universal visitor number that makes every Experiment ready. Use your store’s normal traffic volume as the sanity check.

​
What not to do when reading results

  • Do not end an Experiment in the first two days only because confidence shows 95%. Early confidence can jump around when the sample is small.
  • Do not change the primary metric after the Experiment starts just to make a variant look better.
  • Do not ignore guardrails. A variant can win the primary metric and still hurt profit, order value, or purchase likelihood.

​
Convert the dashboard into a decision

Use this matrix after you check visitor count, run time, and traffic split. Rollout cost includes implementation work, QA, customer support risk, operational changes, and margin or brand tradeoffs. As a default, do not ship a variant for a tiny projected lift if rollout needs theme edits, checkout QA, pricing policy changes, new support macros, or extra fulfillment work. Ship when the gain is clearly worth those changes for your store.

​
Example: Revenue per Visitor winner

The Holiday Pricing Experiment dashboard uses Revenue per Visitor as the primary metric. The Experiment uses a 34/33/33 traffic split across control and two variants. The sample includes 25,540 visitors. Variant B leads. It is up 13.3% in Revenue per Visitor versus control. The estimated monthly revenue uplift is 6,420. The estimated monthly profit uplift is 2,775. Profit is positive, with 13.05% estimated profit uplift. Variant B is a winner when the run time is long enough for your store and the gains beat normal rollout cost.

​
Example: Conversion rate winner but profit loser

Suppose your primary metric is Conversion Rate because you want more visitors to purchase. Variant A increases Conversion Rate by 8%, but Profit per Visitor drops by 12%. The variant used a larger discount, so more visitors bought, but each visitor produced less profit. This is a candidate to ship only if the profit risk is acceptable. In most stores, a better next Experiment would use a smaller discount, a higher free-shipping threshold, or a bundle offer that protects margin.