Kubernetes troubleshooting

Diagnose why Kubernetes workloads fail. Inspect first. Propose a fix. Do not mutate the cluster until the user explicitly approves the change.

Purpose and scope

In scope

• Failed or not-Ready Deployments, ReplicaSets, Pods, Jobs, and CronJobs
• CrashLoopBackOff, ImagePullBackOff, ErrImagePull, OOMKilled, Evicted, Pending
• Failed liveness, readiness, and startup probes
• Resource requests/limits and QoS clues
• Service with no Endpoints, basic kube-dns / CoreDNS, NetworkPolicy deny
• Reading live cluster state with kubectl or reviewing pasted output and manifests

Out of scope

• Writing a greenfield Deployment from scratch (use a deployment Skill)
• Cluster-wide security audits (use a security-audit Skill)
• CNI vendor internals beyond what kubectl and Events show
• Inventing Pod status, logs, or Events that were not collected

When to use

Use this Skill when the user mentions CrashLoopBackOff, ImagePullBackOff, Pending Pods, failed rollouts, 0/1 Ready, OOMKilled, probe failures, Service timeouts, or "the app is down in Kubernetes."

Do not use it for Terraform/AWS account design, CI YAML debugging, or incident command (hand those off). After the cluster is stable, point SRE work at sre-incident-response rather than duplicating incident comms here.

Inputs

Collect at least one of:

• Namespace and workload name
• kubectl access (kubectl get ns succeeds)
• Pasted kubectl describe / logs / Events
• Manifests that produced the failure

If the namespace is unknown, list namespaces and ask. If both live access and pasted output are missing, stop and request them. Do not guess cluster state.

Workflow

Copy and track:

- [ ] 1. Scope the failure
- [ ] 2. Gather read-only evidence
- [ ] 3. Classify the failure mode
- [ ] 4. Confirm with a second signal
- [ ] 5. Recommend a fix with risk
- [ ] 6. Request approval before any write

1. Scope the failure

kubectl config current-context
kubectl get ns
kubectl get deploy,sts,ds,job,cronjob,po,svc,endpoints,events -n <namespace>
kubectl get deploy <name> -n <namespace> -o wide

Identify: namespace, controller, replica counts (DESIRED/CURRENT/READY), image, last Event times. Work from the controller down to Pods, not the other way around.

2. Gather read-only evidence

Prefer the bundled collector when kubectl works:

scripts/collect-pod-diagnostics.sh <namespace> <pod>

Otherwise run:

kubectl get pod <pod> -n <namespace> -o yaml
kubectl describe pod <pod> -n <namespace>
kubectl get events -n <namespace> --field-selector involvedObject.name=<pod> --sort-by='.lastTimestamp'
kubectl logs <pod> -n <namespace> --all-containers --tail=200
kubectl logs <pod> -n <namespace> --previous --all-containers --tail=200
kubectl get rs,ep,endpointslices,svc,netpol -n <namespace>

--previous is mandatory on CrashLoopBackOff; current logs are often empty. Do not add -f, --v=8, or exec unless the user asked for a live session.

If the collector is not present in the agent's Skill directory, run the kubectl commands above; do not fail the diagnosis.

3. Classify the failure mode

Match evidence to references/failure-modes.md. Use this order:

Never pick a class without a quoted Event, container reason, or log line.

4. Confirm with a second signal

• CrashLoop: exit code + --previous logs + probe config
• Image pull: Events Failed to pull image + imagePullSecrets + image string
• OOM: OOMKilled + resources.limits.memory vs observed usage if metrics exist
• Pending: kubectl describe pod scheduling message + node capacity (kubectl describe node only if needed)
• Network: Endpoints, kubectl get svc, CoreDNS Pods, NetworkPolicy — see references/networking.md

If metrics-server is absent, say so. Do not fabricate RAM usage.

5. Recommend a fix

Write a concrete change (manifest snippet, image pin, probe timeout, limit bump) and the verification command. Give blast radius (one Deployment vs node drain). Offer a non-prod dry-run when possible:

kubectl apply --dry-run=server -f <manifest>
kubectl diff -f <manifest>

6. Approval gate

Do not run kubectl apply, kubectl delete, kubectl rollout restart, kubectl scale, kubectl drain, kubectl exec that mutates, or kubectl annotate/label that changes behavior until the user explicitly authorizes that command.

Best practices

• One hypothesis at a time. The first Event is not always the root cause (a pull error can hide a later crash).
• Compare spec.containers[].image to what you expected; floating tags (latest) are a frequent ImagePullBackOff and rollback hazard.
• Treat CrashLoopBackOff as "the process keeps dying," not as a Kubernetes bug.
• For probes: startupProbe for slow boots; never set liveness identical to a heavyweight readiness check that hits a dependent database if that can kill the Pod during dependency blips.
• Requests without limits can overcommit; limits without requests surprise the scheduler. Call this out when OOM or Pending appear.
• Quote timestamps. A 3-day-old Event is not evidence for a current outage.
• Stay inside the given namespace unless Events point at kube-system (CoreDNS, CNI).
• kubectl debug and ephemeral containers are optional and may require extra RBAC; ask first.

Commands and examples

CrashLoopBackOff (example)

kubectl describe pod payments-7f9c4d-xk2n8 -n prod
kubectl logs payments-7f9c4d-xk2n8 -n prod --previous --tail=100

Example evidence (not the user's cluster):

State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1

Previous logs show FATAL: DATABASE_URL is required. Diagnosis: missing env/config, not a scheduler issue. Fix: add the env from a Secret after the user confirms the Secret exists; do not dump Secret values into chat.

ImagePullBackOff (example)

Failed to pull image "ghcr.io/acme/pay:v2.9.1": rpc error: code = Unknown desc = failed to fetch oauth token

Check image spelling, tag, imagePullSecrets, and imagePullPolicy. Do not rotate registry credentials in-cluster without approval.

OOMKilled (example)

Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Limits: memory: 128Mi

Recommend raising the limit or fixing the leak. Do not raise limits blindly in prod; ask for a target and whether HPA/node capacity can absorb it.

Pending (example)

0/6 nodes are available: 3 Insufficient memory, 2 node(s) had taint {dedicated: jobs}, 1 node(s) didn't match Pod's node affinity.

Do not cordon/uncordon or remove taints without approval.

Expected output

# Kubernetes diagnosis

## Summary
- Workload:
- Namespace / context:
- User impact:
- Failure class:
- Confidence (high/medium/low):

## Evidence
- Commands run (exact):
- Controller status:
- Pod status / restart count:
- Container last state / exit code:
- Quoted Events (timestamp + message):
- Quoted log lines (redacted):

## Root cause
What is broken and why, tied to evidence. List assumptions separately.

## Recommended fix
1. Change:
2. Why it works:
3. Risk / blast radius:
4. Verification command:
5. Rollback:

## Approval required
Commands that must not run until the user says yes.

## Follow-ups
Monitoring, probes, resource requests, runbooks.

Fill this only with collected evidence. If a field is unknown, write unknown — not collected instead of guessing.

Error handling

If the collector script fails, print its error and continue with manual kubectl. Never wrap kubectl delete in a "retry" loop.

Security and safety

• Do not print Secret, ConfigMap values that look like keys, cloud credentials, or raw kubeconfig files. Refer to object names only.
• Do not copy kubectl exec output that contains tokens into the report.
• Do not run kubectl delete pod --force --grace-period=0 to "unstick" CrashLoopBackOff; that hides evidence and can worsen load.
• Do not kubectl apply or rollout restart without explicit authorization.
• Do not disable probes or set imagePullPolicy: Never in production as a workaround unless the user accepts the risk in writing.
• Assume least privilege: prefer get/list/watch. Treat edit/patch/delete as gated.
• If the user asks to expose a private Service with a public LoadBalancer to "test," flag the security impact and wait.

Additional resources

• Failure-mode decision tree: references/failure-modes.md
• Service, DNS, and NetworkPolicy: references/networking.md
• Read-only collector: scripts/collect-pod-diagnostics.sh