What is A/B Testing?
A/B testing is a method for making data-driven product decisions by randomly splitting users into groups that see different variants — a control (A) and one or more treatments (B) — and measuring which drives a better outcome on a predefined metric.
Done correctly, A/B testing isolates the impact of a single change by holding everything else constant, so observed differences can be attributed to the variant rather than to chance or confounding factors. This requires a clear hypothesis, a primary metric defined up front, a sufficient sample size, and a test run long enough to reach statistical significance.
PMs rely on experimentation to de-risk decisions, settle internal debates with evidence, and avoid shipping changes that feel right but actually hurt key metrics. Pitfalls to avoid include peeking at results early, testing too many things at once, and ignoring guardrail metrics.
Examples
- A PM tests a new checkout button color and measures conversion; the variant wins by 4% with significance.
- An experiment is stopped because, while clicks rose, a guardrail metric (refund rate) also rose.
Where PMs use this
Related terms
Conversion Rate
The percentage of users who complete a desired action out of those who had the opportunity.
KPI (Key Performance Indicator)
A quantifiable measure used to track progress toward a specific business or product objective.
Feature Flag
A switch that turns functionality on or off in production without deploying new code.
Cohort Analysis
Grouping users by a shared trait (often signup date) to compare how their behavior evolves over time.