Incident Response Best Practices

A condensed summary of the 25 most important incident response practices for Node.js teams - drawn from every page in this section.

Link runbooks from service README: On-call finds rollback steps in 30 seconds - Incident Basics.
Declare severity within 5 minutes: Round up when unsure; downgrade later - Incident Basics.
Assign IC, TL, Comms, Scribe: IC decides; TL debugs - Incident Basics.
First-15-minutes checklist: Deploy correlation, dashboards, logs, blast radius - Incident Basics.
Stabilize before RCA: Rollback, scale, flag-off, or circuit-break - Bad Deploy Rollback.
Log GIT_SHA at startup: Tie logs to exact image digest instantly.
Annotate deploys on Grafana: Vertical line at promote time for correlation.
Separate liveness and readiness: DB down should not kill all pods - Database Outage Mode.
Read-only mode before pool storm: Block writes; pause workers - Database Outage Mode.
OOM triage: heap vs RSS: Different fixes for leak vs buffers - OOM Killer Response.
No heap snapshot under full traffic: Drain pod first; stall makes incident worse.
Rollback API and workers together: Coupled schema needs coupled rollback - Bad Deploy Rollback.
Feature-flag kill before full rollback: When only one path is bad - Bad Deploy Rollback.
Keep last 10 image tags in registry: rollout undo fails if image GC'd.
Plain-language stakeholder updates: Impact, status, next update time - Incident Basics.
Scribe timeline every 10 minutes: Commands, links, decisions with UTC timestamps.
Capture evidence before restart: Memory graphs, pool metrics, trace IDs.
Event-loop lag alert: CPU-low / latency-high is a Node stall pattern - Post-Mortem Template.
Post-mortem within 48 hours for SEV1/SEV2: Blameless, with evidence links - Post-Mortem Template.
Action items with owner and due date: "Be more careful" is not an action item.
Practice rollback quarterly: Game day with scripted rollout undo.
Canary checks p95, not just errors: Latency regressions slip through otherwise.
Circuit breakers on outbound deps: Stop retry amplification during upstream outage.
Incident channel naming convention: [SEV2] service-name short-description.
Share generic learnings platform-wide: Pool alert tuning helps every Postgres consumer.

FAQs

What belongs in a service runbook?

Rollback command, feature-flag kills, dashboard links, escalation path, and recent known issues.

When do we skip a post-mortem?

SEV4 and one-off SEV3 with clear fix and no recurrence risk. Repeated issues require documentation.

How do Node incidents differ from JVM?

Watch event-loop stall vs thread pool exhaustion. Memory is single-process heap plus native buffers.

Incident Basics - getting started
OOM Killer Response - memory triage
Database Outage Mode - degradation
On-Call Runbook Starters - per-symptom runbooks
SLOs and Error Budgets - alert thresholds

Stack versions: This page was written for Node.js 24.18.0 (Active LTS), npm 10+, TypeScript 5.6+, Express 5, Fastify 5, and NestJS 11.

Incident Response Best Practices

A condensed summary of the 25 most important incident response practices for Node.js teams - drawn from every page in this section.

Link runbooks from service README: On-call finds rollback steps in 30 seconds - Incident Basics.

Declare severity within 5 minutes: Round up when unsure; downgrade later - Incident Basics.

Assign IC, TL, Comms, Scribe: IC decides; TL debugs - Incident Basics.

First-15-minutes checklist: Deploy correlation, dashboards, logs, blast radius - Incident Basics.

Stabilize before RCA: Rollback, scale, flag-off, or circuit-break - Bad Deploy Rollback.

Log GIT_SHA at startup: Tie logs to exact image digest instantly.

Annotate deploys on Grafana: Vertical line at promote time for correlation.

Separate liveness and readiness: DB down should not kill all pods - Database Outage Mode.

Read-only mode before pool storm: Block writes; pause workers - Database Outage Mode.

OOM triage: heap vs RSS: Different fixes for leak vs buffers - OOM Killer Response.

No heap snapshot under full traffic: Drain pod first; stall makes incident worse.

Rollback API and workers together: Coupled schema needs coupled rollback - Bad Deploy Rollback.

Feature-flag kill before full rollback: When only one path is bad - Bad Deploy Rollback.

Keep last 10 image tags in registry: rollout undo fails if image GC'd.

Plain-language stakeholder updates: Impact, status, next update time - Incident Basics.

Scribe timeline every 10 minutes: Commands, links, decisions with UTC timestamps.

Capture evidence before restart: Memory graphs, pool metrics, trace IDs.

Event-loop lag alert: CPU-low / latency-high is a Node stall pattern - Post-Mortem Template.

Post-mortem within 48 hours for SEV1/SEV2: Blameless, with evidence links - Post-Mortem Template.

Action items with owner and due date: "Be more careful" is not an action item.

Practice rollback quarterly: Game day with scripted rollout undo.

Canary checks p95, not just errors: Latency regressions slip through otherwise.

Circuit breakers on outbound deps: Stop retry amplification during upstream outage.

Incident channel naming convention: [SEV2] service-name short-description.

Share generic learnings platform-wide: Pool alert tuning helps every Postgres consumer.

FAQs

What belongs in a service runbook?

Rollback command, feature-flag kills, dashboard links, escalation path, and recent known issues.

When do we skip a post-mortem?

SEV4 and one-off SEV3 with clear fix and no recurrence risk. Repeated issues require documentation.

How do Node incidents differ from JVM?

Watch event-loop stall vs thread pool exhaustion. Memory is single-process heap plus native buffers.

Stack versions: This page was written for Node.js 24.18.0 (Active LTS), npm 10+, TypeScript 5.6+, Express 5, Fastify 5, and NestJS 11.

Incident Response Best Practices

FAQs

Related

Buscar en la documentación

Incident Response Best Practices

FAQs

Related