Experiment Archive

Maintaining a searchable archive of every experiment —hypothesis, result, decision, learning —so the accumulated knowledge is available to future team members and the team never re-runs a falsified hypothesis.

Why

Experiments produce learnings, and learnings that are not written down are lost —the team member who ran the experiment leaves, the Slack thread scrolls away, and 6 months later a new team member re-runs the same experiment and wastes a quarter re-learning what was already known. The archive is the institutional memory: every experiment's hypothesis, result, decision, and learning are stored in a searchable format, so a new team member can search 'trial length' and find that EXP-247 tested 7-day vs 14-day, found +15% conversion, and shipped —and they do not need to re-run it. The archive also prevents the 'we already tried that' problem: when someone proposes an experiment, the team searches the archive, finds the prior experiment, and either builds on it (the prior experiment suggested a 10-day trial might work better) or avoids it (the prior experiment falsified the hypothesis). The archive must be searchable (by metric, by feature, by hypothesis keyword), structured (each experiment has the same fields), and linked (experiments that built on or contradicted prior experiments are cross-referenced). The most common failure is no archive —the team runs 200 experiments over 2 years, the learnings live in Slack threads and individual memories, and when the growth lead leaves, 2 years of learning leaves with them. The second failure is an archive that is not searchable —a folder of 200 PDFs that nobody reads because they cannot find what they are looking for.

When

When any experiment concludes (ship, iterate, or abandon). When a team member proposes an experiment (search the archive first). When onboarding a new team member (the archive is their first read). Quarterly to review accumulated learnings.

How

1. Create a structured experiment template with required fields: experiment_id, hypothesis, change, primary_metric, predicted_effect, guardrails, sample_size, start_date, end_date, result, decision, learning, owner, tags —every experiment fills the same template so the archive is uniform and searchable
2. Store the archive in a searchable system (Notion database, Airtable, internal wiki with full-text search) —not a folder of PDFs or a Slack channel, which cannot be searched by metric or hypothesis keyword
3. Add tags to every experiment: the metric (conversion, retention, revenue), the feature (trial, checkout, onboarding), the hypothesis keyword (dilution-of-intent, urgency, friction) —tags make the archive searchable by topic, not just by experiment_id
4. Cross-reference related experiments: if EXP-248 (10-day trial) was built on EXP-247 (7-day trial), link them; if EXP-302 contradicted EXP-247, link them with a 'contradicts' note —the cross-references build the knowledge graph
5. Write the learning in plain language, not jargon: 'shorter trials increase conversion because users who intend to buy do so within 3 days; the 14-day trial dilutes intent by giving them time to forget' —the learning is what a future team member reads, and jargon makes it inaccessible
6. Include the 'what we would do differently' field: every experiment, especially failed ones, should note what the team would do differently next time —this is the meta-learning that improves the experimentation process itself
7. Search the archive before proposing any experiment: 'has anyone tested trial length before?' —if yes, read the prior experiment and either build on it or avoid it; if no, proceed with the new experiment
8. Review the archive quarterly: read all experiments from the quarter, identify patterns (what hypotheses tend to hold, what mechanisms tend to fail), and write a quarterly learnings summary —the summary is the high-level takeaway that onboards new team members
9. Link the archive to the experiment registry: the registry (pre-experiment) and the archive (post-experiment) are two halves of the same system, and the experiment_id links them —this prevents the team from running an experiment without registering it (no archive entry) or registering it without running it (no result)
10. When a team member leaves, have them review their experiments in the archive and add any context that is in their head but not in the archive —the offboarding review is the last chance to capture tacit knowledge before it leaves

Example

# Experiment archive: EXP-247
ID: EXP-247
Date: 2026-09-22 to 2026-10-10
Owner: @growth-pm (jane@ez4code.com)
Status: shipped

Hypothesis:
  If we shorten the trial from 14 to 7 days, trial-to-paid
  conversion will increase by 15% because 80% of conversions
  happen by day 3 and the longer trial dilutes intent.

Change: trial length 14 days -> 7 days (treatment)
Primary metric: trial-to-paid conversion
Predicted effect: +15% (3.2% -> 3.7%), MDE +10%
Guardrails: refund rate, rev/trial, churn, error rate, NPS

Result:
  control:    3.18% (1,221 / 38,400)
  treatment:  3.66% (1,405 / 38,400)
  lift:       +15.1% (95% CI: +8.2% to +22.0%, p=0.002)
  guardrails: all intact
  segments:   new +25%, mobile +38%, desktop +8%, returning +2%

Decision: ship to 100% (gradual rollout 10% -> 50% -> 100%)
Production effect (2-week check): +11.2% (74% of estimate)

Learning:
  Shorter trials increase conversion via reduced dilution of
  intent. Users who intend to buy do so within 3 days; the
  14-day trial gives them 11 more days to forget or churn.
  Effect is larger for new and mobile users (less established
  habits, more urgency-sensitive). Production effect was 74%
  of experimental estimate (winner's curse calibration).

What we would do differently:
  - test 10-day trial as a middle ground (EXP-248)
  - segment the result earlier (we found mobile +38% only
    in post-hoc analysis; pre-register segment hypotheses)
  - measure long-term retention (does faster conversion
    lead to higher or lower 90-day retention?)

Tags: conversion, trial-length, dilution-of-intent, pricing
Related experiments:
  - EXP-248 (10-day trial): built on EXP-247, result +9%
  - EXP-302 (3-day trial): contradicted, result -4% (too short)
  - EXP-180 (trial reminder emails): complementary, +6%

Archive search:
  Search "trial length" -> EXP-247, EXP-248, EXP-302
  Search "conversion" -> 47 experiments
  Search "dilution of intent" -> EXP-247, EXP-248

Quarterly review (Q4 2026):
  Pattern: hypotheses about "reducing friction" tend to hold
  (+8% to +25%), hypotheses about "creating urgency" are mixed
  (EXP-247 +15%, EXP-302 -4%). Mechanism validation is the
  differentiator: friction reduction has a clear data-backed
  mechanism, urgency creation often relies on intuition.

Common Problems

⚠No archive —the team runs 200 experiments over 2 years, the learnings live in Slack threads and individual memories, and when the growth lead leaves, 2 years of learning leaves with them; the archive is the institutional memory that survives team turnover.
⚠An archive that is not searchable —a folder of 200 PDFs that nobody reads because they cannot find what they are looking for; the archive must be in a searchable system (Notion, Airtable, wiki) with tags and full-text search.
⚠Not searching the archive before proposing an experiment —a new team member proposes 'shorten the trial to 7 days', the team runs it, and only afterward discovers EXP-247 already tested it 6 months ago; searching first avoids re-running falsified hypotheses.
⚠Archiving only the wins —the team documents the +15% experiments but not the -4% ones, and the archive becomes a highlight reel; the failures are where the most valuable learnings are (falsified mechanisms), and they must be archived with equal care.
⚠No 'what we would do differently' field —the experiment is archived with the result but not the meta-learning, and the team repeats the same methodological mistakes (not pre-registering segments, not checking guardrails early) in future experiments.

Best Practices

✓Archive every experiment (ship, iterate, and abandon) in a searchable system with a structured template (hypothesis, result, decision, learning, tags) —the archive is the institutional memory, and without it, learnings are lost to team turnover and Slack scroll.
✓Add tags (metric, feature, hypothesis keyword) and cross-references to related experiments —tags make the archive searchable by topic, and cross-references (built-on, contradicted-by) build the knowledge graph that prevents re-running falsified hypotheses.
✓Write the learning in plain language and include the 'what we would do differently' field —the learning is what a future team member reads, and the meta-learning (methodological improvements) is what improves the experimentation process itself.
✓Search the archive before proposing any experiment —if a prior experiment tested the same hypothesis, build on it or avoid it; this prevents the team from wasting a quarter re-learning what is already in the archive.
✓Review the archive quarterly to identify patterns (which hypotheses tend to hold, which mechanisms tend to fail) and write a quarterly learnings summary —the summary onboards new team members and identifies the meta-patterns that improve the team's hypothesis quality over time.