A/B Test
Randomly split users into two groups: one gets version A, one gets version B, then you measure which one wins.
Think of it like
Like a restaurant trying two recipes and seeing which one customers prefer.
Example
Half of users get prompted with 'How can I help?' the other half with 'What do you need?' You measure which phrasing leads to more productive conversations.
How it actually works
A/B testing is the gold standard for measuring real-world impact because it controls for confounds. You're measuring the effect of one change in isolation. The catch: you need enough traffic to detect differences, you need to avoid peeking, and multiple hypothesis testing can fool you. Also, not every decision should be A/B tested (some are too costly, others are clearly better).
For product teams
Run A/B tests before major launches to de-risk decisions and measure real user impact.
For engineers
Implement randomization, logging, and analysis infrastructure; plan sample size before running.
Related
- Human Eval — Alternate evaluation approach.
- Win Rate — What you measure in tests.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome