Skip to content
Back to Idea Validation

Wizard of Oz Test

Users interact with what looks like a fully automated product, but a human does the work behind the curtain.

Why

The Wizard of Oz test lets you deliver a full product experience before you have built the automation. Users see a real UI, submit real requests, and get real results — but you (the wizard) manually fulfill each request behind the scenes. This tests the end-to-end value proposition (does the output actually solve the problem?) without the months of engineering needed to build the real system. It is especially powerful for AI products, where the quality of the output matters more than the speed of generation, and where building the real model is the most expensive part.

When

When the product's value is in the output quality, not the automation, and building the real backend is expensive. Takes 1-2 weeks of manual operation. Ideal for AI/ML products, marketplaces, and personalization features.

How

1. Design the full user-facing flow
   - UI must look 100% real — no "demo" badges or placeholders
   - The user should believe they are using a finished product

2. Set up your "curtain"
   - User submits request → you get a notification (Slack/email)
   - You fulfill it manually using tools you already have
   - You send the result back through the real UI

3. Add realistic delay
   - If the real product would take 30s, do not return in 1s
   - Match expected latency so users judge the real experience

4. Operate for 1-2 weeks with 10-30 users
   - Fulfill every request to the best of your ability
   - Track: completion rate, satisfaction, willingness to pay, retention

5. Learn what to actually build
   - Which parts of your manual process can be automated?
   - Which parts need human judgment (and should stay manual)?
   - What output quality bar do you need to hit?

6. Decide
   - Users love the output → build automation for the repeatable parts
   - Users do not value the output → kill, the problem is the proposition not the tech

Example

Idea: "AI that writes personalized cold sales emails from a prospect's LinkedIn profile."
Wizard of Oz (2 weeks):
  - Built a real UI: paste LinkedIn URL → "generating email" spinner → email output
  - Behind the curtain: I got a Slack ping, researched the profile, wrote the email by hand
  - Added a 45-second delay to feel like AI processing

Results (22 users, 140 emails generated):
  - 18/22 said the emails were "better than what I write myself"
  - 12 asked to connect their CRM (showing workflow integration intent)
  - 6 offered to pay $30/mo on the spot
  - Key learning: users wanted tone customization, not just personalization

Decision: value validated → build real AI, prioritize tone controls.

Common Problems

  • ⚠The manual output quality is higher than what automation can achieve — users are disappointed when the real (automated) product launches and quality drops.
  • ⚠Burning out from manual fulfillment — if you cannot sustain 2 weeks of manual work, scale down the user count rather than letting quality slip.
  • ⚠Users discovering the human behind the curtain — if a request comes back with a typo or a personal aside, the illusion breaks and the test is contaminated.
  • ⚠Not tracking what parts of your manual process are repeatable — without this, you do not know what to automate first when you build the real thing.

Best Practices

  • ✓Keep a structured log of every manual fulfillment: what you did, how long it took, what judgment you applied — this becomes your automation spec.
  • ✓Set a quality floor and refuse to ship below it — better to tell users "high volume, try later" than to send bad output that invalidates the test.
  • ✓Debrief each user in a 15-minute call after the test — ask what they would change, whether they would pay, what workflow they imagine.
  • ✓Plan the transition: decide before you start which manual steps will become AI, which stay human, and which become a hybrid (AI draft + human review).