SRE incident response

Stabilize the user-facing problem first. Communicate on a fixed cadence. Preserve evidence. Do not restart, rollback, fail over, or scale production without explicit authorization.

This Skill ends at incident close and postmortem scheduling. Full write-up belongs in a dedicated postmortem Skill (not bundled in v0.1.0).

Purpose and scope

In scope

• Declaring an incident, severity (SEV), and roles (commander, comms, ops)
• Mitigation vs root cause; error-budget and SLO impact at a high level
• Status updates using the bundled template
• Evidence: timelines, dashboards, logs, deploy markers — collection, not deletion
• Decision records for rollback vs fix-forward
• Handoff notes for the postmortem

Out of scope

• Deep Kubernetes Pod debugging steps (use kubernetes-troubleshooting, then return)
• Terraform apply as the mitigation (use terraform-review for the change, still gated)
• Writing the complete postmortem document
• Pretending to have paged humans, updated Statuspage, or queried Prometheus when you did not

When to use

Use this Skill when the user reports a production outage, SEV, pager alert, SLO burn, "site is down," regional failover, or asks you to run incident command.

Do not use it for a quiet code review or a hypothetical tabletop unless the user asks for a drill (label the session drill so nobody pages).

Inputs

• Symptom and start time (UTC)
• Affected product, region, and customer impact
• Links: dashboards, alerts, deploy pipeline, previous incidents
• Who is already in the channel
• Whether this is prod, staging, or a drill

If impact is unknown, treat it as SEV-2 or worse until proven otherwise, and ask for a customer-facing symptom.

Workflow

- [ ] 1. Declare and classify
- [ ] 2. Staff roles
- [ ] 3. Stabilize (mitigate)
- [ ] 4. Communicate
- [ ] 5. Investigate with evidence
- [ ] 6. Resolve and watch
- [ ] 7. Close and hand off

1. Declare and classify

Say explicitly: "This is an incident" (or "This is a drill"). Assign a working name (INC-YYYYMMDD-service-symptom).

Pick severity using references/severity-and-comms.md:

If data loss or suspected breach, say so immediately; do not wait for a pretty graph. Preserve logs; do not rotate them away.

2. Staff roles

• Incident commander: decisions, timeboxes, approval of mitigations
• Ops/technical lead: executes approved commands
• Comms: customer and stakeholder updates
• Scribe: timeline

If the user is alone, they hold commander + ops; you act as scribe and technical advisor. Do not silently take commander authority over production.

3. Stabilize

Mitigation beats root cause. Typical levers (describe, do not run unless approved):

• Rollback the last deploy
• Feature flag off
• Fail over to a healthy region
• Shed load / scale up (scale-to-zero is usually harmful)
• Disable a crashing watcher (not kubectl delete --force as a first step)

For Kubernetes symptoms, run kubernetes-troubleshooting in parallel for diagnosis, then bring the recommended mitigation back here for approval.

Timebox investigation (5–10 minutes) before the next mitigation decision. If the first rollback failed, say so; do not loop restarts.

Never restart without approval. Never failover without approval. Never run a database migrate as a "quick fix" without approval.

4. Communicate

Post on the cadence in references/severity-and-comms.md. Use assets/incident-status-template.md.

Every update includes: current SEV, user impact, what changed since last update, next check-in time. Do not speculate in customer channels. Internal notes can hold hypotheses labeled unconfirmed.

5. Investigate with evidence

Build a timeline of facts:

HH:MMZ  source  fact

Allowed evidence: alert timestamps, deploy IDs, error-rate screenshots the user pasted, kubectl/logs output actually collected, SLO burn from a system they queried.

Forbidden: invented Prometheus values, fake "p99 is 2.3s", imaginary Git SHAs.

Capture:

• Alert name and threshold
• First and last deploy in the window
• Change in error rate / latency / saturation (if measured)
• Correlation: "errors started T+4m after deploy abc123" is a hypothesis until confirmed

Do not clear queues, truncate tables, or kubectl delete evidence (failed Pods, Evicted objects) until logs are exported.

6. Resolve and watch

An incident is not resolved when a graph blips green once. Define:

• Mitigation that is in place
• Monitoring window (e.g. 2× the alert for interval, minimum 15 minutes for SEV-1/2)
• Owner watching
• Customer message that the impact ended

If the error budget for the window is already exhausted, say so; do not hide it.

7. Close and hand off

Produce:

• Final status (SEV, duration, impact)
• Timeline
• Mitigations applied (and which were approved)
• Open questions
• Postmortem owner, due date (SEV-1/2: start within 24–48h)

Do not write the five-whys novel here. Link the next Skill/process.

Best practices

• One change at a time so you can tell what helped.
• Prefer rollback of a known-bad deploy over speculative config edits.
• Keep a single source of truth channel; side DMs fragment the timeline.
• SLI/SLO: name the SLI (availability, latency, freshness) and whether you are guessing.
• Customer comms: impact and workaround, not internal hostname lore.
• After SEV-1, freeze non-essential deploys until commander lifts the freeze.
• Blameless language in the timeline ("the deploy increased error rate"), not ("Alice broke prod").

Commands and examples

Commands depend on the stack. Always print the command and wait.

Example — confirm impact (read-only):

# illustration only; use the user's actual dashboard/CLI
kubectl get deploy -n prod
kubectl get events -n prod --sort-by='.lastTimestamp' | tail

Example — proposed mitigation (blocked until approval):

# DO NOT RUN unless the user explicitly approves
# kubectl rollout undo deploy/payments -n prod

Example — incident brief the Skill expects:

Start: 2026-11-11 16:40Z
Symptom: checkout HTTP 5xx ~40%
Last deploy: payments v2.9.1 at 16:28Z
Alert: HighErrorRate prod-payments

Work this as SEV-2 until proven smaller. Hypothesis: bad deploy. Next step: confirm error traces, then ask for approval to rollback.

Expected output

# Incident INC-...

## Declaration
- SEV:
- Drill: yes/no
- Commander / ops / comms / scribe:
- Start (UTC):
- Affected SLI/SLO:

## User impact
What customers cannot do, regions, workarounds.

## Current status
Mitigating | Monitoring | Resolved

## Timeline (UTC)
- HH:MM  fact (source)

## Hypotheses
- unconfirmed: ...

## Mitigations
| Action | Approved by | Time | Result |
| --- | --- | --- | --- |

## Evidence gathered
Links, commands, redacted snippets.

## Communications
Last customer/internal update + next scheduled update.

## Approval required
Pending production actions (restart, rollback, failover, apply).

## Close / handoff
Duration, remaining risk, postmortem owner and due date, open questions.

Error handling

Security and safety

• Never restart without approval. Never failover without approval. Never terraform apply as a silent mitigation.
• Do not dump secrets, tokens, heap dumps with PII, or customer records into chat. Redact.
• Do not disable auth, TLS, or NetworkPolicy "to unblock" without calling the security impact and waiting.
• Do not delete failed Pods, log groups, or snapshots that are evidence.
• Do not page random people or claim that a page went out unless the user confirms the paging tool result.
• Production-impacting database commands (DROP, TRUNCATE, restore) are always gated.

Additional resources

• Severity and communication cadence: references/severity-and-comms.md
• Status update template: assets/incident-status-template.md