Reliability Target

Set reliability targets per service tier so investment matches the service's criticality to the business.

Why

Not every service deserves 99.99% reliability — that costs 10x more than 99.9% and is wasted on an internal admin tool. Treating every service the same either over-invests in unimportant services (wasted engineering time) or under-invests in critical ones (outages that cost revenue). A tiered model (Tier 1 = revenue-critical = 99.95%, Tier 2 = user-facing = 99.9%, Tier 3 = internal = 99.5%) lets you spend reliability effort where it matters and set appropriate SLOs, alerting, and on-call coverage per tier. This is how mature organizations scale reliability: not by making everything 100%, but by matching the reliability investment to the business impact of each service.

When

When you have 3+ services with different criticalities. Revisit when a service's business role changes (e.g., internal tool becomes user-facing).

How

1. Define 3-4 service tiers
   - Tier 1 (critical): revenue-direct or auth — outage = immediate revenue loss
   - Tier 2 (important): user-facing features — outage = degraded UX, no immediate revenue loss
   - Tier 3 (internal): admin tools, dashboards — outage = developer friction, no user impact
   - Tier 4 (experimental): non-prod, sandboxes — best-effort only

2. Assign a reliability target per tier
   - Tier 1: 99.95% (22m downtime/28d), 24/7 on-call, SEV1 paging
   - Tier 2: 99.9% (43m/28d), business-hours on-call, SEV2 paging
   - Tier 3: 99.5% (3.4h/28d), fix next business day, ticket only
   - Tier 4: best-effort, no SLO, no on-call

3. Tag every service with its tier
   - The tier drives: SLO, alerting sensitivity, on-call coverage, postmortem requirement
   - Store in service catalog (e.g., Backstage) so it's discoverable

4. Match investment to tier
   - Tier 1: redundancy, multi-region, canary, full APM, runbook per alert
   - Tier 2: single-region + canary, APM, runbook per alert
   - Tier 3: single-region, basic monitoring, runbook per alert
   - Don't gold-plate Tier 3; don't under-build Tier 1

5. Review tier assignments annually
   - A Tier 3 that's now used by customers → promote to Tier 2
   - A Tier 1 that's been deprecated → demote to Tier 3

Example

# service-catalog.yaml — tier assignments drive reliability investment
services:
  - name: checkout-api
    tier: 1
    slo: 0.9995
    oncall: 24/7
    regions: [us-east, us-west, eu]  # multi-region for Tier 1
    alerts: [SEV1 page, SEV2 page]
  - name: search-api
    tier: 2
    slo: 0.999
    oncall: business-hours
    regions: [us-east]
    alerts: [SEV2 page, SEV3 ticket]
  - name: admin-dashboard
    tier: 3
    slo: 0.995
    oncall: none
    regions: [us-east]
    alerts: [SEV3 ticket]

# Investment per tier (engineering hours/quarter):
#   Tier 1: 200h (redundancy, canary, runbooks, multi-region failover drills)
#   Tier 2: 80h  (canary, runbooks)
#   Tier 3: 20h  (basic monitoring, runbooks)
# This matches spend to business impact instead of gold-plating everything.

Common Problems

⚠Every service is Tier 1 — when everything is critical, nothing is, and reliability effort is spread too thin to actually protect the revenue-critical services.
⚠Tier assigned once and never revisited — an internal tool that became customer-facing stays Tier 3 and causes a user-facing outage with no on-call.
⚠Tier drives SLO but not investment — Tier 1 has a 99.95% target but no multi-region or canary, so the target is aspirational, not achievable.
⚠No service catalog — tiers live in someone's head, so new services aren't tagged and inherit the default (usually over- or under-built).

Best Practices

✓Define 3-4 tiers with explicit reliability targets, on-call coverage, and investment levels per tier — match spend to business impact, not to uniform 100%.
✓Store tier assignments in a service catalog (Backstage/internal) so every service is tagged and new ones inherit the right defaults.
✓Make the tier drive investment, not just SLO — Tier 1 gets multi-region, canary, and failover drills; Tier 3 gets single-region and basic monitoring.
✓Review tier assignments annually; promote services that became user-facing and demote deprecated ones so the model stays accurate.