Incident Response Best Practices
A condensed summary of the 25 most important incident response practices for Node.js teams - drawn from every page in this section.
//
Search across all documentation pages
A condensed summary of the 25 most important incident response practices for Node.js teams - drawn from every page in this section.
Link runbooks from service README: On-call finds rollback steps in 30 seconds - Incident Basics.
Declare severity within 5 minutes: Round up when unsure; downgrade later - Incident Basics.
Assign IC, TL, Comms, Scribe: IC decides; TL debugs - Incident Basics.
First-15-minutes checklist: Deploy correlation, dashboards, logs, blast radius - Incident Basics.
Stabilize before RCA: Rollback, scale, flag-off, or circuit-break - Bad Deploy Rollback.
Log GIT_SHA at startup: Tie logs to exact image digest instantly.
Annotate deploys on Grafana: Vertical line at promote time for correlation.
Separate liveness and readiness: DB down should not kill all pods - Database Outage Mode.
Read-only mode before pool storm: Block writes; pause workers - Database Outage Mode.
OOM triage: heap vs RSS: Different fixes for leak vs buffers - OOM Killer Response.
No heap snapshot under full traffic: Drain pod first; stall makes incident worse.
Rollback API and workers together: Coupled schema needs coupled rollback - Bad Deploy Rollback.
Feature-flag kill before full rollback: When only one path is bad - Bad Deploy Rollback.
Keep last 10 image tags in registry: rollout undo fails if image GC'd.
Plain-language stakeholder updates: Impact, status, next update time - Incident Basics.
Scribe timeline every 10 minutes: Commands, links, decisions with UTC timestamps.
Capture evidence before restart: Memory graphs, pool metrics, trace IDs.
Event-loop lag alert: CPU-low / latency-high is a Node stall pattern - Post-Mortem Template.
Post-mortem within 48 hours for SEV1/SEV2: Blameless, with evidence links - Post-Mortem Template.
Action items with owner and due date: "Be more careful" is not an action item.
Practice rollback quarterly: Game day with scripted rollout undo.
Canary checks p95, not just errors: Latency regressions slip through otherwise.
Circuit breakers on outbound deps: Stop retry amplification during upstream outage.
Incident channel naming convention: [SEV2] service-name short-description.
Share generic learnings platform-wide: Pool alert tuning helps every Postgres consumer.
Rollback command, feature-flag kills, dashboard links, escalation path, and recent known issues.
SEV4 and one-off SEV3 with clear fix and no recurrence risk. Repeated issues require documentation.
Watch event-loop stall vs thread pool exhaustion. Memory is single-process heap plus native buffers.
Stack versions: This page was written for Node.js 24.18.0 (Active LTS), npm 10+, TypeScript 5.6+, Express 5, Fastify 5, and NestJS 11.
Reviewed by Chris St. John·Last updated Jul 16, 2026