Deploy Debugging Playbook

A failed or misbehaving deploy is a race between your diagnosis and your users' patience. This playbook structures the first thirty minutes: stabilize first, diagnose second, and fix forward or roll back by decision rather than reflex. Its core discipline is separating "what changed" from "what broke" — in deploys, the thing that changed is the prime suspect but not always the culprit.

When to use this skill

• A deploy failed mid-rollout, or completed and then errors climbed
• The new version works except for one endpoint, one region, or one subset of users
• A deploy "succeeded" but the change is invisibly absent in production
• Post-rollback, to find the actual cause before the retry

Workflow

1. Stabilize before understanding. First decision, inside the first minutes: is user impact ongoing and material? If yes and rollback is safe (no irreversible migration mid-flight), roll back now and diagnose from the stable state. Debugging live on users is a choice — make it consciously and briefly, never by default.
2. Fix the timeline. Establish with timestamps: when the deploy started and finished per instance, when the first bad signal appeared, and what else changed in the window — config, flags, dependencies, data jobs, upstream releases. The deploy is the loudest change, not necessarily the only one.
3. Confirm what is actually running. Get the live version identifier from the serving instances themselves, not from the deploy tool's opinion. Split-brain fleets — half old, half new — explain most "intermittent" post-deploy behavior. Check every replica set, region, and cache layer that could pin the old build.
4. Localize by contrast. What distinguishes failing requests from succeeding ones? New-version instances versus stragglers, one route versus all, authenticated versus anonymous, one shard versus the rest. The differential is the diagnosis; collect it before reading any code.
5. Read the boot sequence when instances fail to start: config parse, secret fetch, migration gate, port bind, health-check grace period. Startup failures are usually environmental — the code runs fine somewhere configured differently.
6. Check the compatibility seams when errors are partial: schema versus code version skew, cache entries serialized by the old build, in-flight jobs enqueued with old payload shapes, clients pinned to a removed response field. Deploys fail at the seams between old state and new code.
7. Decide: forward or back. Fix forward only when the cause is identified, the fix is smaller than the rollback, and you can verify it within one cycle. Otherwise roll back, verify recovery against the signal from step 2, and take the diagnosis offline.
8. Write the two-paragraph debrief while it is fresh: cause, detection gap, and the one guard (check, alert, gate) that would have caught this before rollout. File it where the team looks.

Decision points

• Roll back immediately when: error rate or latency is user-visible and climbing, the cause is unknown, and the rollback path is verified safe.
• Hold and diagnose when: impact is contained (one flag, one canary slice) and the evidence is perishable (in-memory state that a rollback destroys).
• Never roll back blind when: a schema migration ran — verify the old code tolerates the new schema first, or you trade one outage for a worse one.

Guardrails

• Every claim gets a timestamp and a source; "it started around the deploy" is not a timeline.
• Verify version identity from the running process before debugging code — minutes spent here save hours spent debugging software that is not actually deployed.
• One mutation at a time while diagnosing; simultaneous fixes destroy the experiment.
• If two deploys are suspect, bisect by environment rather than by argument.
• The debrief is part of the playbook, not optional paperwork. An undocumented deploy failure will be re-run by a colleague within the quarter.