Skip to content
Back to Growth & Acquisition

Experiment Framework

Standardize how experiments are proposed, prioritized, run, and learned from, so the team compounds knowledge instead of repeating work.

Why

Without a framework, experiments are ad hoc — someone runs a test, the result lives in a Slack thread, and 6 months later someone else re-runs the same test because the learning was lost. A framework (experiment template, prioritization score, central log, learning archive) turns each experiment into a permanent asset: the hypothesis, result, and learning are searchable, so the team builds on what worked rather than rediscovering it. The prioritization score (ICE: Impact × Confidence × Ease) prevents the team from spending 3 weeks on a low-impact experiment while a high-impact one waits. The framework also enforces statistical rigor (pre-registered sample size, stop condition) that prevents the false positives that pollute the learning archive. The output of the framework is not wins — it is a playbook of confirmed hypotheses that any new team member can execute.

When

When the team runs 2+ experiments/month. Adopt the framework before scaling experimentation; retrofitting is painful.

How

1. Create an experiment template: hypothesis, primary metric, guardrails, predicted effect, sample size, stop date, owner
2. Set up a central experiment log (Notion, Airtable, or a repo) where every experiment is registered BEFORE it runs
3. Prioritize with ICE score: Impact (1-10, how much will it move the metric?) × Confidence (1-10, how sure are we?) × Ease (1-10, how fast to ship?)
4. Run experiments in priority order; never run more than 2-3 concurrently per team (context-switching kills throughput)
5. At the stop date, classify: won (effect confirmed), lost (effect refuted), inconclusive (underpowered or no effect)
6. Write a 3-sentence learning for EVERY experiment: what we tested, what happened, what it tells us about users
7. Add confirmed hypotheses to the playbook (searchable, tagged by funnel stage and metric); archive refuted ones so they're not re-attempted
8. Review the framework quarterly: are experiments shipping on time? is the learning archive used? are false positives under control?

Example

# Experiment template (pre-registered)
Hypothesis: "Adding a progress bar to onboarding step 2 will lift
  activation by 5% ± 2%, because users currently don't know how
  many steps remain and drop off."
Primary metric: activation_rate  |  Guardrails: signup_conv, time_to_aha
Predicted effect: +5%  |  Sample size: 6,000/arm  |  Stop: 2026-11-15
ICE score: 7 × 6 × 8 = 336  |  Owner: @alice

# Central log (ICE-prioritized queue)
Rank | Experiment              | ICE  | Status
1    | Onboarding progress bar | 336  | running
2    | Referral trigger on save| 280  | next
3    | Pricing page A/B        | 252  | queued
4    | Email subject test      | 90   | queued (low impact)

# Learning archive (searchable, tagged)
Tag: [activation, onboarding] "Progress bar on step 2: +4.2% activation
  [CI +2.1 to +6.3]. Users drop because they don't see the end.
  → Playbook: always show progress in multi-step flows."

# Quarterly review
Experiments shipped: 14 (target 12) ✓
Learning archive lookups: 23 (team is using it) ✓
False positives: 1 (under control) ✓

Common Problems

  • ⚠No central log, so experiments live in Slack threads and the learning is lost; 6 months later the same test is re-run.
  • ⚠Prioritizing by gut or by who shouts loudest, so the team spends 3 weeks on a low-impact test while a high-impact one waits.
  • ⚠No learning written for inconclusive experiments; 'inconclusive' is a result, and the reason (underpowered vs no effect) is learning.
  • ⚠Running 10 experiments concurrently per team; context-switching kills throughput and the results are low-quality.

Best Practices

  • ✓Pre-register every experiment in a central log with hypothesis, sample size, and stop date; this is what makes the learning permanent.
  • ✓Prioritize with ICE (Impact × Confidence × Ease); it prevents 3 weeks on a low-impact test while a high-impact one waits.
  • ✓Write a 3-sentence learning for every experiment including inconclusive; the reason for inconclusiveness is itself learning.
  • ✓Cap concurrent experiments at 2-3 per team; context-switching kills throughput and degrades every running experiment.