OOM Killer Response

Q: What exit code means OOMKilled?

137 (128 + 9 SIGKILL). Kubernetes events show OOMKilled as the reason.

Q: How do I take a heap snapshot in production?

Use kill -USR2 if wired to heapdump, or exec into a drained pod. Never snapshot the only replica under full traffic.

When Kubernetes marks your Node.js pod OOMKilled, you have minutes to decide: leak, traffic spike, or misconfigured limits. This page triages heap vs RSS and picks the right mitigation.

Recipe

Quick-reference recipe card - copy-paste ready.

# 1. Confirm OOMKilled (not CrashLoop for other reasons)
kubectl describe pod <pod> -n production | grep -E "OOMKilled|Reason|Exit Code"
 
# 2. Check memory metrics: heap vs RSS
# Grafana: process_resident_memory_bytes vs v8 heap used
 
# 3. Correlate with deploy and traffic
kubectl rollout history deployment/<svc>
# Compare RPS spike to memory slope
 
# 4. Mitigate
kubectl scale deployment/<svc> --replicas=<N+2>   # traffic spike
kubectl rollout undo deployment/<svc>               # bad deploy
# leak: throttle traffic, rolling restart, schedule heap snapshot

When to reach for this:

Pods restart with OOMKilled and Exit Code 137
Memory graph climbs steadily under flat traffic (leak suspect)
Memory spikes with RPS but recovers after scale-out (capacity suspect)
RSS grows while heapUsed stays flat (native buffers, gRPC, sharp)

Working Example

// src/observability/memory-metrics.ts
import { monitorEventLoopDelay } from "node:perf_hooks";
import v8 from "node:v8";
 
const loopDelay = monitorEventLoopDelay({ resolution: 20 });
loopDelay.enable();
 
export function startMemoryReporter(log: (obj: object) => void, intervalMs = 30_000) {
  setInterval(() => {
    const mem = process.memoryUsage();
    const heapStats = v8.getHeapStatistics();
    log({
      msg: "memory_sample",
      rssMb: Math.round(mem.rss / 1024 / 1024),
      heapUsedMb: Math.round(mem.heapUsed / 1024 / 1024),
      heapTotalMb: Math.round(mem.heapTotal / 1024 / 1024),
      externalMb: Math.round(mem.external / 1024 / 1024),
      arrayBuffersMb: Math.round(mem.arrayBuffers / 1024 / 1024),
      heapLimitMb: Math.round(heapStats.heap_size_limit / 1024 / 1024),
      eventLoopP99Ms: Math.round(loopDelay.percentile(99) / 1e6),
    });
    loopDelay.reset();
  }, intervalMs).unref();
}

What this demonstrates:

rss vs heapUsed divergence signals native/buffer growth
external and arrayBuffers track Buffer and TypedArray pressure
Event-loop p99 alongside memory catches GC pause spirals

Deep Dive

Heap vs RSS

Metric	What it measures	Grows when
`heapUsed`	V8 JavaScript objects	Leaked closures, growing Maps, cached objects
`rss`	Total process resident memory	Heap + native addons + thread stacks + mmap
`external`	C++ objects bound to JS	Buffers, native crypto, some ORM drivers
`arrayBuffers`	Detached ArrayBuffer memory	Binary payloads, file uploads

Leak vs Traffic Spike

Signal	Likely leak	Likely traffic/capacity
Traffic	Flat RPS	RPS up 2-10x
Memory slope	Linear over hours	Step change at spike
Restarts	Same pod, same pattern	All pods, proportional load
`heapUsed`	Tracks RSS up	May stay moderate
Fix	Code fix + snapshot	Scale HPA, tune pool, raise limits

Triage Decision Tree

OOMKilled?
  ├─ Deploy in last 60 min? → rollback first
  ├─ RPS spike? → scale replicas, check pool sizes
  ├─ heapUsed ≈ RSS climbing? → leak hunt (snapshot, clinic heapprofiler)
  ├─ RSS >> heapUsed? → check Buffers, sharp, gRPC, Prisma engine
  └─ Limits too tight? → raise memory request/limit with evidence

Kubernetes Limits

resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "768Mi"   # headroom above steady-state RSS

Set NODE_OPTIONS=--max-old-space-size=512 below k8s limit (leave ~30% for non-heap)
OOMKilled at 768Mi with 700Mi RSS means limit, not necessarily leak
Use VPA recommendations only after observing steady-state under load

Gotchas

Snapshot under load - heapdump stalls the event loop. Fix: Copy pod, drain traffic, snapshot on the clone.
Ignoring external - Leak detectors watch heap only. Fix: Log arrayBuffers and external every 30s.
Restart without evidence - Masks leak rate. Fix: Scribe records memory graph URL before restart.
Same limit for API and worker - Workers hold larger job payloads. Fix: Separate Deployments and limits.
Global caches without TTL - Map keyed by userId grows forever. Fix: LRU with max entries or Redis.

Alternatives

Alternative	Use When	Don't Use When
`clinic heapprofiler`	Reproducible leak in staging	Production under customer load
`clinic doctor`	Event-loop + memory combined	You already know it is OOM
Raise limits only	Proven capacity spike	RSS climbs under flat traffic
Switch to worker threads	CPU-bound blocking loop	I/O-bound API (won't fix leak)

FAQs

What exit code means OOMKilled?

137 (128 + 9 SIGKILL). Kubernetes events show OOMKilled as the reason.

How do I take a heap snapshot in production?

Use kill -USR2 <pid> if wired to heapdump, or exec into a drained pod. Never snapshot the only replica under full traffic.

Can connection pool leaks cause OOM?

They cause exhaustion and timeouts before OOM, but idle pool growth increases RSS. Check pg pool totalCount vs max.

Should I disable GC with --expose-gc?

Only in controlled diagnostics. Forcing GC hides leak signals in metrics.

Incident Basics - first-15-minutes checklist
Memory and GC Tuning - proactive tuning
Clinic.js Suite - leak profiling
Connection Pool Tuning - pool sizing
HPA and Resource Limits - k8s limits

NodeSource Blog & Resources - Enterprise Node.js performance and security content.
Node.js Best Practices (GitHub) - Curated production best practices.
Node.js GitHub Repository - Source code, issues, contribution guides.
Node.js Design Patterns, 4th Edition - Production-grade application patterns.
Distributed Systems with Node.js - Enterprise-ready distributed systems.

Stack versions: This page was written for Node.js 24.18.0 (Active LTS), npm 10+, TypeScript 5.6+, Express 5, Fastify 5, and NestJS 11.

Recipe

Quick-reference recipe card - copy-paste ready.

# 1. Confirm OOMKilled (not CrashLoop for other reasons)
kubectl describe pod <pod> -n production | grep -E "OOMKilled|Reason|Exit Code"
 
# 2. Check memory metrics: heap vs RSS
# Grafana: process_resident_memory_bytes vs v8 heap used
 
# 3. Correlate with deploy and traffic
kubectl rollout history deployment/<svc>
# Compare RPS spike to memory slope
 
# 4. Mitigate
kubectl scale deployment/<svc> --replicas=<N+2>   # traffic spike
kubectl rollout undo deployment/<svc>               # bad deploy
# leak: throttle traffic, rolling restart, schedule heap snapshot

When to reach for this:

Pods restart with OOMKilled and Exit Code 137
Memory graph climbs steadily under flat traffic (leak suspect)
Memory spikes with RPS but recovers after scale-out (capacity suspect)
RSS grows while heapUsed stays flat (native buffers, gRPC, sharp)

Working Example

// src/observability/memory-metrics.ts
import { monitorEventLoopDelay } from "node:perf_hooks";
import v8 from "node:v8";
 
const loopDelay = monitorEventLoopDelay({ resolution: 20 });
loopDelay.enable();
 
export function startMemoryReporter(log: (obj: object) => void, intervalMs = 30_000) {
  setInterval(() => {
    const mem = process.memoryUsage();
    const heapStats = v8.getHeapStatistics();
    log({
      msg: "memory_sample",
      rssMb: Math.round(mem.rss / 1024 / 1024),
      heapUsedMb: Math.round(mem.heapUsed / 1024 / 1024),
      heapTotalMb: Math.round(mem.heapTotal / 1024 / 1024),
      externalMb: Math.round(mem.external / 1024 / 1024),
      arrayBuffersMb: Math.round(mem.arrayBuffers / 1024 / 1024),
      heapLimitMb: Math.round(heapStats.heap_size_limit / 1024 / 1024),
      eventLoopP99Ms: Math.round(loopDelay.percentile(99) / 1e6),
    });
    loopDelay.reset();
  }, intervalMs).unref();
}

What this demonstrates:

rss vs heapUsed divergence signals native/buffer growth
external and arrayBuffers track Buffer and TypedArray pressure
Event-loop p99 alongside memory catches GC pause spirals

Deep Dive

Heap vs RSS

Metric	What it measures	Grows when
`heapUsed`	V8 JavaScript objects	Leaked closures, growing Maps, cached objects
`rss`	Total process resident memory	Heap + native addons + thread stacks + mmap
`external`	C++ objects bound to JS	Buffers, native crypto, some ORM drivers
`arrayBuffers`	Detached ArrayBuffer memory	Binary payloads, file uploads

Leak vs Traffic Spike

Signal	Likely leak	Likely traffic/capacity
Traffic	Flat RPS	RPS up 2-10x
Memory slope	Linear over hours	Step change at spike
Restarts	Same pod, same pattern	All pods, proportional load
`heapUsed`	Tracks RSS up	May stay moderate
Fix	Code fix + snapshot	Scale HPA, tune pool, raise limits

Triage Decision Tree

OOMKilled?
  ├─ Deploy in last 60 min? → rollback first
  ├─ RPS spike? → scale replicas, check pool sizes
  ├─ heapUsed ≈ RSS climbing? → leak hunt (snapshot, clinic heapprofiler)
  ├─ RSS >> heapUsed? → check Buffers, sharp, gRPC, Prisma engine
  └─ Limits too tight? → raise memory request/limit with evidence

Kubernetes Limits

resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "768Mi"   # headroom above steady-state RSS

Set NODE_OPTIONS=--max-old-space-size=512 below k8s limit (leave ~30% for non-heap)
OOMKilled at 768Mi with 700Mi RSS means limit, not necessarily leak
Use VPA recommendations only after observing steady-state under load

Gotchas

Snapshot under load - heapdump stalls the event loop. Fix: Copy pod, drain traffic, snapshot on the clone.
Ignoring external - Leak detectors watch heap only. Fix: Log arrayBuffers and external every 30s.
Restart without evidence - Masks leak rate. Fix: Scribe records memory graph URL before restart.
Same limit for API and worker - Workers hold larger job payloads. Fix: Separate Deployments and limits.
Global caches without TTL - Map keyed by userId grows forever. Fix: LRU with max entries or Redis.

Alternatives

Alternative	Use When	Don't Use When
`clinic heapprofiler`	Reproducible leak in staging	Production under customer load
`clinic doctor`	Event-loop + memory combined	You already know it is OOM
Raise limits only	Proven capacity spike	RSS climbs under flat traffic
Switch to worker threads	CPU-bound blocking loop	I/O-bound API (won't fix leak)

FAQs

What exit code means OOMKilled?

137 (128 + 9 SIGKILL). Kubernetes events show OOMKilled as the reason.

How do I take a heap snapshot in production?

Use kill -USR2 <pid> if wired to heapdump, or exec into a drained pod. Never snapshot the only replica under full traffic.

Can connection pool leaks cause OOM?

They cause exhaustion and timeouts before OOM, but idle pool growth increases RSS. Check pg pool totalCount vs max.

Should I disable GC with --expose-gc?

Only in controlled diagnostics. Forcing GC hides leak signals in metrics.

Incident Basics - first-15-minutes checklist
Memory and GC Tuning - proactive tuning
Clinic.js Suite - leak profiling
Connection Pool Tuning - pool sizing
HPA and Resource Limits - k8s limits

NodeSource Blog & Resources - Enterprise Node.js performance and security content.
Node.js Best Practices (GitHub) - Curated production best practices.
Node.js GitHub Repository - Source code, issues, contribution guides.
Node.js Design Patterns, 4th Edition - Production-grade application patterns.
Distributed Systems with Node.js - Enterprise-ready distributed systems.

Stack versions: This page was written for Node.js 24.18.0 (Active LTS), npm 10+, TypeScript 5.6+, Express 5, Fastify 5, and NestJS 11.

OOM Killer Response

Search Documentation

OOM Killer Response