Incident Basics
8 examples to run your first Node.js incident calmly - 6 basic and 2 intermediate. Covers severity, communication, and the first-15-minutes checklist.
Prerequisites
- On-call rotation with PagerDuty/Opsgenie wired to SLO alerts
- Service README links to runbooks (see On-Call Runbook Starters)
- Structured logs with
requestIdandtraceId(see Request Correlation IDs) - Dashboards for p95 latency, error rate, heap RSS, and DB pool wait time
Basic Examples
1. Declare Severity Within 5 Minutes
| Severity | Definition | Node example |
|---|---|---|
| SEV1 | Customer-facing outage or data loss risk | All pods OOMKilled, 100% 5xx |
| SEV2 | Major degradation, workaround exists | p95 > 5s, checkout slow |
| SEV3 | Minor impact, next business day fix | One tenant webhook delay |
| SEV4 | Cosmetic or internal-only | Staging deploy failed |
- SEV1 triggers Incident Commander + status page update within 15 minutes
- When unsure, round up - downgrading is easier than under-responding
- Document severity in the incident channel topic:
[SEV2] orders-api high latency
Related: Incident Response Best Practices - full runbook checklist
2. Assign Incident Roles
Incident Commander (IC) - owns decisions, timeline, escalation
Technical Lead (TL) - drives mitigation, reads logs/metrics
Comms Lead - status page, stakeholder updates
Scribe - timeline, commands run, links to graphs- IC does not debug alone - delegates investigation to TL
- Scribe posts every 10 minutes in
#incident-orders-api - Rotate IC if incident exceeds 2 hours
3. First-15-Minutes Node Checklist
[ ] Acknowledge page; join bridge / Slack huddle
[ ] Confirm blast radius (which services, tenants, regions)
[ ] Check last deploy (git SHA, time, who)
[ ] Grafana: error rate, p95, pod restarts, OOMKilled count
[ ] Logs: spike in 5xx, ECONNREFUSED, pool timeout, ETIMEDOUT
[ ] Recent config/secret/flag changes
[ ] Decide: rollback, scale, feature-flag kill, or investigate
[ ] Post first external update if SEV1/SEV2- Do not restart pods blindly before checking deploy correlation
- Capture
kubectl get pods/ ECS task state before changes - Save heap snapshot only if memory is suspect and traffic is throttled
Related: OOM Killer Response - memory triage | Bad Deploy Rollback - rollback paths
4. Stabilize Before Root Cause
Mitigation beats perfect diagnosis in the first hour.
| Symptom | Fast mitigation |
|---|---|
| Bad deploy | Roll back to previous image SHA |
| Traffic spike | Scale HPA max; enable rate limit |
| DB pool exhausted | Reduce worker concurrency; pause non-critical queues |
| Memory leak | Rolling restart + traffic shift to healthy pods |
| Upstream 502 | Open circuit breaker; serve cached/degraded response |
- Log every mitigation action with timestamp in the incident thread
- Mark "stabilized" when error rate returns below SLO for 15 minutes
- Root cause analysis continues after customer impact stops
5. Communicate in Plain Language
Bad: "Event loop blocked due to libuv saturation."
Good: "Checkout is failing for about 30% of users. We deployed a fix at 14:22 UTC and errors are dropping. Next update in 20 minutes."
Template for stakeholder updates:
**Impact:** <who is affected, what is broken>
**Status:** Investigating | Mitigating | Monitoring | Resolved
**Actions:** <what we did in last 20 min>
**Next update:** <time UTC>- PM and support need customer impact, not heap dumps
- Link internal dashboards only in engineering channels
6. Gather Node-Specific Evidence
# Pod memory and restarts
kubectl top pods -n production -l app=orders-api
kubectl describe pod <pod> | grep -A5 "Last State"
# Recent deploy
kubectl rollout history deployment/orders-api -n production
# Logs with correlation
# filter: service=orders-api level=error last 30m- Export p95 latency and error-rate graphs covering deploy window
- Note Node version, image digest, and
NODE_OPTIONSif memory-related - Save slow-query logs if DB errors appear in stack traces
Intermediate Examples
7. Bridge Call Structure (30-Minute Cadence)
0:00 IC states severity, impact, current hypothesis
0:05 TL shares latest metric snapshot
0:10 Decision: rollback / scale / flag off / continue debug
0:15 Comms posts external update
0:25 Scribe reads timeline; confirm action owners
0:30 Repeat or downgrade severity- Cap attendees at 8 - move deep debugging to a side thread
- End bridge when SEV1/SEV2 is mitigated; hand off to follow-up for RCA
8. Hand Off to Post-Mortem
When resolved:
[ ] Mark incident resolved in PagerDuty
[ ] Post final customer-facing summary
[ ] Create post-mortem doc within 48 hours (SEV1/SEV2)
[ ] List action items with owners and due dates
[ ] No blame - focus on systems and process gaps- Use Post-Mortem Template for SEV1/SEV2
- Track action items in Jira/Linear with
incident-followuplabel
FAQs
Who should be Incident Commander?
On-call engineer by default. Escalate to tech lead if blast radius crosses multiple services or exceeds 1 hour at SEV1.
When do we page the whole team?
SEV1 only, or SEV2 lasting more than 30 minutes without a mitigation path. Avoid paging for known maintenance.
Should we restart Node pods during an incident?
Yes if OOMKilled or unresponsive, but capture logs and note deploy correlation first. Prefer rolling restart over deleting all pods at once.
How long until we write a post-mortem?
Draft within 48 hours for SEV1/SEV2. Review in team meeting within 5 business days.
Related
- OOM Killer Response - memory incidents
- Bad Deploy Rollback - deploy rollback
- Post-Mortem Template - RCA template
- On-Call Runbook Starters - service runbooks
- Incident Response Best Practices - condensed checklist
Stack versions: This page was written for Node.js 24.18.0 (Active LTS), npm 10+, TypeScript 5.6+, Express 5, Fastify 5, and NestJS 11.