Incident Basics

8 examples to run your first Node.js incident calmly - 6 basic and 2 intermediate. Covers severity, communication, and the first-15-minutes checklist.

Prerequisites

On-call rotation with PagerDuty/Opsgenie wired to SLO alerts
Service README links to runbooks (see On-Call Runbook Starters)
Structured logs with requestId and traceId (see Request Correlation IDs)
Dashboards for p95 latency, error rate, heap RSS, and DB pool wait time

Basic Examples

1. Declare Severity Within 5 Minutes

Severity	Definition	Node example
SEV1	Customer-facing outage or data loss risk	All pods OOMKilled, 100% 5xx
SEV2	Major degradation, workaround exists	p95 > 5s, checkout slow
SEV3	Minor impact, next business day fix	One tenant webhook delay
SEV4	Cosmetic or internal-only	Staging deploy failed

SEV1 triggers Incident Commander + status page update within 15 minutes
When unsure, round up - downgrading is easier than under-responding
Document severity in the incident channel topic: [SEV2] orders-api high latency

Related: Incident Response Best Practices - full runbook checklist

2. Assign Incident Roles

Incident Commander (IC)  - owns decisions, timeline, escalation
Technical Lead (TL)      - drives mitigation, reads logs/metrics
Comms Lead               - status page, stakeholder updates
Scribe                   - timeline, commands run, links to graphs

IC does not debug alone - delegates investigation to TL
Scribe posts every 10 minutes in #incident-orders-api
Rotate IC if incident exceeds 2 hours

3. First-15-Minutes Node Checklist

[ ] Acknowledge page; join bridge / Slack huddle
[ ] Confirm blast radius (which services, tenants, regions)
[ ] Check last deploy (git SHA, time, who)
[ ] Grafana: error rate, p95, pod restarts, OOMKilled count
[ ] Logs: spike in 5xx, ECONNREFUSED, pool timeout, ETIMEDOUT
[ ] Recent config/secret/flag changes
[ ] Decide: rollback, scale, feature-flag kill, or investigate
[ ] Post first external update if SEV1/SEV2

Do not restart pods blindly before checking deploy correlation
Capture kubectl get pods / ECS task state before changes
Save heap snapshot only if memory is suspect and traffic is throttled

Related: OOM Killer Response - memory triage | Bad Deploy Rollback - rollback paths

4. Stabilize Before Root Cause

Mitigation beats perfect diagnosis in the first hour.

Symptom	Fast mitigation
Bad deploy	Roll back to previous image SHA
Traffic spike	Scale HPA max; enable rate limit
DB pool exhausted	Reduce worker concurrency; pause non-critical queues
Memory leak	Rolling restart + traffic shift to healthy pods
Upstream 502	Open circuit breaker; serve cached/degraded response

Log every mitigation action with timestamp in the incident thread
Mark "stabilized" when error rate returns below SLO for 15 minutes
Root cause analysis continues after customer impact stops

5. Communicate in Plain Language

Bad: "Event loop blocked due to libuv saturation."

Good: "Checkout is failing for about 30% of users. We deployed a fix at 14:22 UTC and errors are dropping. Next update in 20 minutes."

Template for stakeholder updates:

**Impact:** <who is affected, what is broken>
**Status:** Investigating | Mitigating | Monitoring | Resolved
**Actions:** <what we did in last 20 min>
**Next update:** <time UTC>

PM and support need customer impact, not heap dumps
Link internal dashboards only in engineering channels

6. Gather Node-Specific Evidence

# Pod memory and restarts
kubectl top pods -n production -l app=orders-api
kubectl describe pod <pod> | grep -A5 "Last State"
 
# Recent deploy
kubectl rollout history deployment/orders-api -n production
 
# Logs with correlation
# filter: service=orders-api level=error last 30m

Export p95 latency and error-rate graphs covering deploy window
Note Node version, image digest, and NODE_OPTIONS if memory-related
Save slow-query logs if DB errors appear in stack traces

Intermediate Examples

7. Bridge Call Structure (30-Minute Cadence)

0:00  IC states severity, impact, current hypothesis
0:05  TL shares latest metric snapshot
0:10  Decision: rollback / scale / flag off / continue debug
0:15  Comms posts external update
0:25  Scribe reads timeline; confirm action owners
0:30  Repeat or downgrade severity

Cap attendees at 8 - move deep debugging to a side thread
End bridge when SEV1/SEV2 is mitigated; hand off to follow-up for RCA

8. Hand Off to Post-Mortem

When resolved:

[ ] Mark incident resolved in PagerDuty
[ ] Post final customer-facing summary
[ ] Create post-mortem doc within 48 hours (SEV1/SEV2)
[ ] List action items with owners and due dates
[ ] No blame - focus on systems and process gaps

Use Post-Mortem Template for SEV1/SEV2
Track action items in Jira/Linear with incident-followup label

FAQs

Who should be Incident Commander?

On-call engineer by default. Escalate to tech lead if blast radius crosses multiple services or exceeds 1 hour at SEV1.

When do we page the whole team?

SEV1 only, or SEV2 lasting more than 30 minutes without a mitigation path. Avoid paging for known maintenance.

Should we restart Node pods during an incident?

Yes if OOMKilled or unresponsive, but capture logs and note deploy correlation first. Prefer rolling restart over deleting all pods at once.

How long until we write a post-mortem?

Draft within 48 hours for SEV1/SEV2. Review in team meeting within 5 business days.

OOM Killer Response - memory incidents
Bad Deploy Rollback - deploy rollback
Post-Mortem Template - RCA template
On-Call Runbook Starters - service runbooks
Incident Response Best Practices - condensed checklist

Node.js Best Practices (GitHub) - Curated production best practices.
NodeSource Blog & Resources - Enterprise Node.js performance and security content.
DigitalOcean Node.js Tutorials - Production-oriented how-tos.
Distributed Systems with Node.js - Enterprise-ready distributed systems.
Node.js Cookbook, 5th Edition - Practical server-side recipes.

Stack versions: This page was written for Node.js 24.18.0 (Active LTS), npm 10+, TypeScript 5.6+, Express 5, Fastify 5, and NestJS 11.

Prerequisites

On-call rotation with PagerDuty/Opsgenie wired to SLO alerts
Service README links to runbooks (see On-Call Runbook Starters)
Structured logs with requestId and traceId (see Request Correlation IDs)
Dashboards for p95 latency, error rate, heap RSS, and DB pool wait time

Basic Examples

1. Declare Severity Within 5 Minutes

Severity	Definition	Node example
SEV1	Customer-facing outage or data loss risk	All pods OOMKilled, 100% 5xx
SEV2	Major degradation, workaround exists	p95 > 5s, checkout slow
SEV3	Minor impact, next business day fix	One tenant webhook delay
SEV4	Cosmetic or internal-only	Staging deploy failed

SEV1 triggers Incident Commander + status page update within 15 minutes
When unsure, round up - downgrading is easier than under-responding
Document severity in the incident channel topic: [SEV2] orders-api high latency

Related: Incident Response Best Practices - full runbook checklist

2. Assign Incident Roles

Incident Commander (IC)  - owns decisions, timeline, escalation
Technical Lead (TL)      - drives mitigation, reads logs/metrics
Comms Lead               - status page, stakeholder updates
Scribe                   - timeline, commands run, links to graphs

IC does not debug alone - delegates investigation to TL
Scribe posts every 10 minutes in #incident-orders-api
Rotate IC if incident exceeds 2 hours

3. First-15-Minutes Node Checklist

[ ] Acknowledge page; join bridge / Slack huddle
[ ] Confirm blast radius (which services, tenants, regions)
[ ] Check last deploy (git SHA, time, who)
[ ] Grafana: error rate, p95, pod restarts, OOMKilled count
[ ] Logs: spike in 5xx, ECONNREFUSED, pool timeout, ETIMEDOUT
[ ] Recent config/secret/flag changes
[ ] Decide: rollback, scale, feature-flag kill, or investigate
[ ] Post first external update if SEV1/SEV2

Do not restart pods blindly before checking deploy correlation
Capture kubectl get pods / ECS task state before changes
Save heap snapshot only if memory is suspect and traffic is throttled

Related: OOM Killer Response - memory triage | Bad Deploy Rollback - rollback paths

4. Stabilize Before Root Cause

Mitigation beats perfect diagnosis in the first hour.

Symptom	Fast mitigation
Bad deploy	Roll back to previous image SHA
Traffic spike	Scale HPA max; enable rate limit
DB pool exhausted	Reduce worker concurrency; pause non-critical queues
Memory leak	Rolling restart + traffic shift to healthy pods
Upstream 502	Open circuit breaker; serve cached/degraded response

Log every mitigation action with timestamp in the incident thread
Mark "stabilized" when error rate returns below SLO for 15 minutes
Root cause analysis continues after customer impact stops

5. Communicate in Plain Language

Bad: "Event loop blocked due to libuv saturation."

Good: "Checkout is failing for about 30% of users. We deployed a fix at 14:22 UTC and errors are dropping. Next update in 20 minutes."

Template for stakeholder updates:

**Impact:** <who is affected, what is broken>
**Status:** Investigating | Mitigating | Monitoring | Resolved
**Actions:** <what we did in last 20 min>
**Next update:** <time UTC>

PM and support need customer impact, not heap dumps
Link internal dashboards only in engineering channels

6. Gather Node-Specific Evidence

# Pod memory and restarts
kubectl top pods -n production -l app=orders-api
kubectl describe pod <pod> | grep -A5 "Last State"
 
# Recent deploy
kubectl rollout history deployment/orders-api -n production
 
# Logs with correlation
# filter: service=orders-api level=error last 30m

Export p95 latency and error-rate graphs covering deploy window
Note Node version, image digest, and NODE_OPTIONS if memory-related
Save slow-query logs if DB errors appear in stack traces

Intermediate Examples

7. Bridge Call Structure (30-Minute Cadence)

0:00  IC states severity, impact, current hypothesis
0:05  TL shares latest metric snapshot
0:10  Decision: rollback / scale / flag off / continue debug
0:15  Comms posts external update
0:25  Scribe reads timeline; confirm action owners
0:30  Repeat or downgrade severity

Cap attendees at 8 - move deep debugging to a side thread
End bridge when SEV1/SEV2 is mitigated; hand off to follow-up for RCA

8. Hand Off to Post-Mortem

When resolved:

[ ] Mark incident resolved in PagerDuty
[ ] Post final customer-facing summary
[ ] Create post-mortem doc within 48 hours (SEV1/SEV2)
[ ] List action items with owners and due dates
[ ] No blame - focus on systems and process gaps

Use Post-Mortem Template for SEV1/SEV2
Track action items in Jira/Linear with incident-followup label

FAQs

Who should be Incident Commander?

On-call engineer by default. Escalate to tech lead if blast radius crosses multiple services or exceeds 1 hour at SEV1.

When do we page the whole team?

SEV1 only, or SEV2 lasting more than 30 minutes without a mitigation path. Avoid paging for known maintenance.

Should we restart Node pods during an incident?

Yes if OOMKilled or unresponsive, but capture logs and note deploy correlation first. Prefer rolling restart over deleting all pods at once.

How long until we write a post-mortem?

Draft within 48 hours for SEV1/SEV2. Review in team meeting within 5 business days.

OOM Killer Response - memory incidents
Bad Deploy Rollback - deploy rollback
Post-Mortem Template - RCA template
On-Call Runbook Starters - service runbooks
Incident Response Best Practices - condensed checklist

Node.js Best Practices (GitHub) - Curated production best practices.
NodeSource Blog & Resources - Enterprise Node.js performance and security content.
DigitalOcean Node.js Tutorials - Production-oriented how-tos.
Distributed Systems with Node.js - Enterprise-ready distributed systems.
Node.js Cookbook, 5th Edition - Practical server-side recipes.

Stack versions: This page was written for Node.js 24.18.0 (Active LTS), npm 10+, TypeScript 5.6+, Express 5, Fastify 5, and NestJS 11.

Incident Basics

Search Documentation

Incident Basics