Test Design

How to design an experiment that isolates the effect of one change —hypothesis, metric, allocation, duration —before any traffic flows.

Why

An A/B test is a question asked of your users, and the design is the phrasing of the question. A poorly designed test asks a confused question and gets an uninterpretable answer. The three most common design failures are: testing two changes at once (you cannot attribute the lift to either), testing on a metric that is not affected by the change (you will see noise and conclude 'no effect'), and stopping early when the result looks good (you inflate false positives and ship noise). A well-designed test states a falsifiable hypothesis ('changing the CTA color from grey to blue will increase click-through rate by at least 5%'), defines one primary metric that the change is expected to move, allocates traffic 50/50 to eliminate allocation bias, and pre-commits a sample size and duration so the decision rule is fixed before there is a result to be tempted by. This discipline is what makes the test's answer trustworthy.

When

When you have a specific change, a hypothesis about its effect, and enough traffic to reach the required sample size. Not when you 'want to try something'.

How

1. State the hypothesis as a falsifiable statement: "Changing X from A to B will increase [primary metric] by at least [MDE]." —if you cannot state it this precisely, you are not ready to test
2. Define one primary metric —the metric the change is designed to move. If you cannot name it, the change is not specific enough to test
3. Define 2-5 guardrail metrics —metrics the change must not harm (e.g., page load time, error rate, revenue). These are the 'do no harm' checks
4. Compute the required sample size from the baseline metric rate, the MDE, alpha (0.05), and power (0.80) —use an online calculator or stats library
5. Compute the required duration: sample size / daily traffic. If duration > 4 weeks, either increase MDE, reduce the number of variants, or do not run the test (traffic is too low)
6. Allocate traffic 50/50 (or evenly across variants) using a consistent hash of user_id —never use time-based splitting (before/after) because it confounds the change with time
7. Pre-commit the design in the experiment registry: hypothesis, primary metric, guardrails, sample size, duration, decision rule —before any traffic is exposed
8. Define the decision rule: "Ship if primary metric lifts >= MDE with p<0.05 AND no guardrail degrades by > X%." —the rule is fixed before the result exists

Example

# Experiment: CTA color grey -> blue
Registered: 2026-09-20  Owner: @pm-checkout

Hypothesis: changing the checkout CTA from grey (#6b7280) to blue
(#2563eb) will increase click-through rate by at least 5% relative.

Primary metric: CTA click-through rate (clicks / impressions)
  Baseline: 12.4%  MDE: 5% relative (= 0.62pp absolute)

Guardrail metrics:
  - page load time (must not increase > 100ms)
  - checkout error rate (must not increase > 0.5pp)
  - overall conversion (must not decrease > 1%)

Sample size: 38,400 impressions per arm (two-proportion z-test,
alpha=0.05, power=0.80, baseline=0.124, MDE=0.0062)
Duration: 38400 / 12000 daily impressions = 3.2 days -> round to 4 days
Allocation: 50/50 via hash(user_id) % 2, sticky per user

Decision rule:
  Ship if CTR lift >= 5% relative AND p < 0.05
       AND page load increase < 100ms
       AND error rate increase < 0.5pp
       AND overall conversion decrease < 1%
  Kill if any guardrail breaches OR CTR lift < 0 (significant decrease)

Pre-committed in experiment-registry #447. No peeking before day 4.

Common Problems

⚠Testing multiple changes in one variant —'redesign the checkout page' bundles 15 changes, and a 10% lift cannot be attributed to any of them, so you cannot iterate; you ship the whole bundle or nothing, and the learning is zero.
⚠Using before/after (time-based) comparison instead of concurrent A/B —anything that changed between Tuesday and Wednesday (a holiday, a competitor outage, a marketing email) is confounded with your change, and you cannot separate them.
⚠Not pre-committing the decision rule —without a fixed rule, the team decides 'ship' after seeing the result, and the threshold silently moves to whatever makes the favored outcome significant (p-hacking by intuition).
⚠Choosing an MDE after seeing the result —'the lift was 3%, so let's call 3% our MDE and declare significance' is circular reasoning that turns every test into a win regardless of the underlying truth.

Best Practices

✓Test one change at a time —if you have 15 changes, run 15 sequential tests or a multi-armed bandit; never bundle them into one variant, because the learning is the point and a bundle teaches you nothing.
✓Use concurrent allocation (hash-based, sticky per user) never time-based —concurrent eliminates time-confounders; before/after includes every other thing that changed in the period.
✓Pre-commit the full design (hypothesis, metric, MDE, sample size, decision rule) in a registry before exposure —this is what makes p-hacking impossible, because the rule is fixed before the result exists.
✓Choose the MDE based on business value, not statistical convenience —'what lift would make this worth shipping?' is the right question; 'what lift can we detect in 3 days?' is the wrong one, and it leads to shipping trivially small effects.