Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.abconvert.io/llms.txt

Use this file to discover all available pages before exploring further.

This page helps you answer one practical question: “Can I trust this result yet?”
Analytics v2 gives you two statistical views of the same experiment:
  • Frequentist analysis for clear pass/fail decision support.
  • Bayesian analysis for early directional guidance when traffic is still low.
Use both together. They are most useful when they point in the same direction.

​
Frequentist analysis: clear decision support

Frequentist analysis is useful when you need a strict rule for go or no-go decisions. In Analytics v2, this view is represented mainly by p-value and confidence interval.

​
How to read p-value

P-valueWhat it means
> 0.10Weak evidence. Usually too early or no clear difference.
0.05 - 0.10Borderline. Keep running the experiment.
< 0.05Statistically significant. Strong evidence of a real difference.
< 0.01Very strong evidence.
A p-value below 0.05 does not mean there is a 95% chance your variant is better. It means the observed gap is unlikely to be random noise.

​
How to read confidence intervals

Confidence intervals show the likely range of the true metric value. A narrow range means a more stable estimate. A wide range means more uncertainty. For example, if Revenue per Visitor is 2.40 and the interval is [1.10, 3.70], the true value could reasonably fall anywhere in that range.
Statistical significance table comparing Control and Variant A with uplift percentages and value ranges for each group

​
Bayesian analysis: early directional guidance

Bayesian analysis is useful when you do not have enough traffic for a strong frequentist result yet. It helps you understand which variant is leading right now and how stable that lead looks.
  • Prob. Beat Control: the probability that a variant is better than control for the selected metric.
  • Prob. Be Best: the probability that a variant is the best among all variants in the experiment.
The advantage is speed of guidance. You can follow the trend before p-value reaches a strict threshold.
Performance overview chart on the Win chance tab showing Variant A probability trend for Conversion Rate over several dates
Do not decide based on one snapshot. Look for a consistent pattern across multiple refreshes.
Follow this order when you review experiment results:
1

Start with Bayesian direction

Check Prob. Beat Control and Prob. Be Best. If they are unstable or keep flipping, keep running the experiment.
2

Confirm with frequentist evidence

Before final rollout, verify that p-value is below 0.05 and confidence intervals are not too wide.
3

Check business sanity

Make sure uplift is meaningful for your store, not just statistically detectable.
4

Decide and ship

Launch the winning variant when Bayesian trend and frequentist evidence both support the same direction.
Run your experiment for at least one full business cycle before making a final decision. For most stores, that means around 1-2 weeks so weekday and weekend behavior are both included.
Avoid stopping too early. Early signals can reverse as more data arrives.

​
What to do when results are inconclusive

Sometimes the result is still unclear after several days. That is normal. If frequentist and Bayesian signals are not aligned yet, do this:
  • Keep the current winner (usually control) while you collect more data.
  • Extend the run window and review again after more traffic arrives.
  • Check segments such as country, device type, or visitor type to find hidden differences.
  • If signals stay flat, treat it as “no meaningful difference” and test a bigger idea next.
Inconclusive results are still useful. They tell you your current variant is not strong enough to justify a rollout yet.