How to Build an Incident Response Runbook for On-Call Engineers

An incident response runbook is a short, alert-specific document that tells an on-call engineer exactly what to check, what to do, and who to involve when something breaks. To build one, start from a real alert, write the triage and mitigation steps as copy-pasteable commands with expected outputs, define clear severity and escalation rules, link the runbook directly from the page, and then test it in drills and update it after every incident. The rest of this post walks through each step, with a template you can adapt.

If you have been paged at 3 a.m. for a service you have never touched, you already know the problem. The alert says something is wrong. It does not say what to do. Good runbooks close that gap, and they reduce the dependence on whichever senior engineer happens to remember how the system works.

What an Incident Response Runbook Is (and Is Not)

We find it helps to be strict about definitions, because vague runbooks usually come from vague scope.

• A runbook is operational. It covers a specific symptom or alert and gives concrete actions.
• A runbook is not architecture documentation. Link to design docs, but do not paste them in. Nobody reads a ten-page system overview mid-incident.
• A runbook is not a full incident management process. Your incident management process defines roles, severity, communication, and post-incident review across the organization. Individual runbooks plug into that process.

The Google SRE book makes a similar distinction. It treats incident management (roles, command, communication) as a separate discipline from the technical playbooks responders use, and it notes that recorded best practices in a playbook meaningfully improve mean time to repair compared to improvising (Google SRE Book, "Introduction" and "Managing Incidents").

Before You Write: Decide Scope

Map runbooks to alerts

The most reliable rule we know: every paging alert should have a runbook, and every runbook should be reachable from an alert. If an alert has no sensible action attached, question whether it should page anyone at all. The SRE book's guidance on on-call is direct about this: pages should be actionable, and alerts that are not actionable create fatigue (Google SRE Book, "Being On-Call").

Start by exporting your current paging alerts and sorting them by how often they fired in the last quarter. Write runbooks for the noisiest and most severe ones first.

Define severity levels once

Severity should be defined in your central incident management process, not reinvented in each runbook. A simple scheme works:

Each runbook then simply states how to classify the specific symptom into one of these levels.

The Anatomy of an Incident Response Runbook

A useful runbook has a predictable shape. When every runbook looks the same, engineers stop hunting for information and start acting. Here is the structure we recommend:

1. Header metadata: service, owning team, alert name, last reviewed date.
2. Summary: one or two sentences on what the alert means in plain language.
3. Impact and severity: who is affected and how to classify it.
4. Triage: quick checks to confirm the problem and narrow the cause.
5. Mitigation: actions that stop the bleeding, ordered from safest to riskiest.
6. Escalation: who to call, and when.
7. Verification: how to confirm the system has recovered.
8. Follow-up: links to dashboards, logs, related runbooks, and the post-incident template.

A runbook template you can copy

# Runbook: CheckoutAPIHighErrorRate

**Service:** checkout-api
**Owner:** payments-platform team (#payments-oncall)
**Alert source:** Prometheus / Alertmanager
**Last reviewed:** YYYY-MM-DD   **Reviewer:** <name>

## Summary
More than 5% of checkout-api requests are returning 5xx for 5+ minutes.
Customers may be unable to complete purchases.

## Severity
- SEV1 if error rate > 25% or checkout success dashboard shows a drop
- SEV2 otherwise

## Triage (5 minutes max)
1. Check recent deploys: `kubectl rollout history deploy/checkout-api -n payments`
2. Check pod health: `kubectl get pods -n payments -l app=checkout-api`
3. Check dependency status: payment-gateway dashboard <link>

## Mitigation
1. If a deploy happened in the last hour: roll back (see command below).
2. If pods are crashlooping from OOM: scale up replicas, then investigate.
3. If payment-gateway is degraded: enable fallback flag `checkout.gateway_fallback`.

## Escalation
- Not mitigated in 15 min: page payments-platform secondary.
- Suspected data integrity issue: page incident commander, declare SEV1.

## Verification
- Error rate below 1% for 10 minutes on <dashboard link>
- Synthetic checkout test passing

## Links
- Dashboard: <link>   Logs: <link>   Design doc: <link>
- Post-incident template: <link>

Writing Steps On-Call Engineers Can Actually Follow

Write for the worst moment

Assume the reader is tired, new to the service, and under pressure. That means short sentences, numbered steps, and no assumed context. If a step says "check the usual dashboard," it has failed.

Make commands copy-pasteable

Every command should run as written, with namespaces and labels filled in. Avoid placeholders where you can, and clearly mark the ones you cannot avoid.

# Roll back checkout-api to the previous revision
kubectl rollout undo deploy/checkout-api -n payments

# Watch the rollout until it completes
kubectl rollout status deploy/checkout-api -n payments --timeout=5m

State the expected output

A command without an expected result forces the engineer to interpret. Add a line like: "Healthy output shows all pods Running with READY 1/1. If you see CrashLoopBackOff, go to Mitigation step 2." This turns the runbook into a decision tree rather than a list of suggestions.

Separate diagnosis from mitigation

Under pressure, people tend to investigate root cause while customers are still affected. Make the order explicit: mitigate first, investigate later. Rolling back a bad deploy is almost always preferable to debugging it live. Mark any destructive or irreversible action in bold with a warning, and say who must approve it.

Wire Runbooks into the Paging Path

A runbook that lives in a wiki nobody can find is not a runbook. Put the link in the alert itself. With Prometheus, the common convention is a runbook_url annotation, which Alertmanager can pass through to your paging tool:

groups:
  - name: checkout-api
    rules:
      - alert: CheckoutAPIHighErrorRate
        expr: |
          sum(rate(http_requests_total{job="checkout-api",code=~"5.."}[5m]))
            /
          sum(rate(http_requests_total{job="checkout-api"}[5m])) > 0.05
        for: 5m
        labels:
          severity: page
          team: payments-platform
        annotations:
          summary: "checkout-api 5xx rate above 5%"
          runbook_url: "https://runbooks.internal/checkout-api/high-error-rate"

We also recommend storing runbooks in version control next to the service code or alert definitions. Changes then go through review, and you can add a CI check that fails if a paging alert is missing a runbook_url.

Roles and Communication

Runbooks handle the technical response. The broader incident management process handles coordination. At minimum, your process should define:

• Incident commander: owns decisions and keeps the response organized.
• Operations lead: executes the technical work, often the on-call engineer.
• Communications lead: updates stakeholders and status pages.

These roles map closely to the Incident Command System approach described in the SRE book and in PagerDuty's open incident response documentation. For security incidents specifically, NIST SP 800-61 provides guidance on integrating incident response into broader risk management. Each runbook's escalation section should point to these roles rather than to named individuals, since people change teams.

Test and Maintain Your Runbooks

Runbooks decay. Commands change, dashboards move, services get renamed. Build maintenance into normal work:

• Update after every incident. Make "review and update the runbook" a standard action item in your post-incident review.
• Run drills. Game days or tabletop exercises, where someone unfamiliar with the service follows the runbook against a staging failure, expose gaps quickly.
• Assign ownership and review dates. A runbook without an owner is a runbook nobody trusts. Flag anything not reviewed in the last two quarters.
• Collect feedback from responders. Add a "this runbook was unclear" link that opens a ticket against the owning team.

A Practical Rollout Plan

If you are starting from nothing, avoid trying to document everything at once:

1. Week 1: Agree on severity levels, roles, and the runbook template.
2. Week 2: Write runbooks for your five highest-volume or highest-severity alerts.
3. Week 3: Add runbook_url to those alerts and a CI check for new ones.
4. Week 4: Run a drill with an engineer who did not write the runbooks.
5. Ongoing: Expand coverage alert by alert and update after every incident.

If you want help standing up on-call rotations, alerting, and runbook practices, our managed services and SRE capabilities cover this work, and the same approach adapts to regulated environments described on our industries page.

Further Reading

• Google, Site Reliability Engineering: "Managing Incidents" and "Being On-Call"
• PagerDuty Incident Response Documentation
• NIST SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management
• Prometheus Alerting Rules documentation

FAQ

What is the difference between a runbook and a playbook?

The terms are often used interchangeably. When teams distinguish them, a runbook usually covers a specific alert or task with step-by-step commands, while a playbook describes a broader response strategy, such as how to handle a security breach or a regional outage, across multiple teams.

How long should an incident response runbook be?

Short enough to scan in under a minute. Most effective runbooks fit on one screen for triage and mitigation, with links out to deeper material. If a runbook grows very long, it is often covering several distinct failure modes and should be split.

Who should write runbooks?

The team that owns the service should write and maintain its runbooks, since they understand its failure modes. Have someone outside the team review or test each runbook, because the author's assumptions are the hardest gaps to spot.

Should runbooks be automated?

Automate steps that are safe, frequent, and well understood, such as restarting a stateless worker or rolling back a deploy. Keep a human decision point for risky or irreversible actions. A runbook that documents the automation, and what to do when it fails, is still valuable.

How often should runbooks be reviewed?

Review a runbook after any incident that used it, and on a regular schedule otherwise. Quarterly or semiannual reviews are common. The key is that each runbook has a named owner and a visible last-reviewed date.