Toolgeist › Math & Unit
Bayesian A/B Test Calculator — Probability to Beat & Expected Loss
A p-value tells you how surprising your data would be if the two variants were identical. It does not tell you the probability that B is better — in the frequentist framework that quantity does not exist. This calculator gives you both numbers that actually drive the decision: the probability B beats A, and the expected loss, which is what you should expect to give up if you ship the wrong variant, measured in percentage points of conversion rate. Posteriors are real Beta(1 + conversions, 1 + non-conversions) distributions, the probability is computed by numerical integration of the regularized incomplete beta, and the result was cross-checked against a 200,000-sample Monte Carlo simulation.
How to use Bayesian A/B Test Calculator — Probability to Beat & Expected Loss
- Enter conversions and visitors for the control and the challenger.
- Read the probability that B beats A, and the expected relative lift.
- Check expected loss — the cost of being wrong, in percentage points.
- Set a loss threshold you can live with and ship when the test drops below it.
Frequently asked questions
- How is this different from a significance calculator?
- A significance test answers "how surprising would this data be if there were no difference?" This answers "what is the probability B is better, and what does it cost me if I am wrong?" Those are different questions, and the second one is the one you actually act on. Both are legitimate; this site has a two-proportion z-test as well, and the two complement each other.
- What exactly is expected loss?
- The average amount of conversion rate you would give up by shipping your chosen variant, averaged over everything you still do not know. It combines the probability of being wrong with the magnitude of being wrong. A 92% win probability with a trivial downside is a fine reason to ship; a 92% win probability where the 8% case is catastrophic is not. Expected loss collapses both into one number in units you care about.
- What prior does it use?
- A uniform Beta(1,1) prior, which treats every conversion rate between 0 and 100% as equally plausible before seeing data. It is the standard non-committal choice and its influence disappears quickly — with a few hundred visitors the data dominates entirely. With very small samples you will see the posterior mean pulled slightly toward 50%; that is the prior being honest about how little the data says.
- Can I stop the test as soon as it looks good?
- More safely than with a p-value, but not carelessly. Repeatedly peeking at a frequentist test and stopping at p < 0.05 badly inflates false positives. Expected loss is a statement about your current posterior rather than about a sampling procedure, so monitoring it continuously is far more defensible — but a small sample yields wide posteriors and a large expected loss, which is the tool telling you to keep going.
- Why is the probability 50% when both variants have identical numbers?
- Because the two posteriors are then identical, so B is above A exactly half the time. That is the correct answer, and it is a useful sanity check: any calculator that reports something other than 50% for identical inputs is broken.
- Does this handle revenue or only conversion rates?
- Only binary conversion rates — each visitor either converted or did not. Revenue per visitor is continuous and heavy-tailed, so it needs a different model entirely; applying a Beta-Binomial to it would give confidently wrong answers.
Related Math & Unit tools
Also searched for: bayesian ab test calculator · probability b beats a · expected loss ab testing · beta binomial conversion rate · when to stop an ab test
Browse all 738 free tools on Toolgeist →