Runtime Ops Highlights Summary

Every highlight bullet from the 7 pages in this section, gathered on one page and grouped by the page it came from.

The Node.js Process Lifecycle

• A Node process moves through a lifecycle - start, ready, serving, draining, exit - not just on and off
• SIGTERM is a request to stop, not a kill - the process decides how long draining takes
• Liveness and readiness are different questions asked by different parts of the platform
• The event loop keeps a process alive exactly as long as it has open handles or pending work

Runtime Ops Basics

• Treat Node as a long-lived server process supervised by the platform
• Handle SIGTERM: stop accepting, drain HTTP, close DB pools
• Document uncaughtException and unhandledRejection policy per service
• Expose /health and /ready for orchestrator probes
• Log structured JSON with request IDs to stdout
• Set NODE_OPTIONS heap limits from load test RSS data

Graceful Shutdown

• Stop accepting requests and drain connections on SIGTERM before exit
• Call server.close() to stop accepting requests and finish in-flight work
• Use pool.end() to cleanly drain database connections during shutdown
• Set app timeout less than pod terminationGracePeriodSeconds to force exit
• Fail readiness probes during drain to stop load balancers sending traffic
• Track background jobs separately, server.close() does not wait for them

Zero-Downtime Deploys

• Combine rolling deployments, readiness draining, and graceful shutdown to avoid downtime
• Set maxUnavailable: 0 to maintain full capacity during rolling pod replacement
• New pods must pass /ready checks before endpoints route traffic to them
• Use expand-contract pattern: deploy compatible code before schema migrations
• Liveness probes should check health only, never databases, to prevent restart loops
• Add preStop sleep and server.close() to gracefully drain in-flight requests on shutdown

PM2 & systemd

• PM2 cluster mode enables zero-downtime reload with instances on single VMs
• Match systemd TimeoutStopSec to app shutdown time for consistent graceful stop
• Use symlink atomic deploys to prevent half-written releases during swap
• Never run PM2 in Docker or K8s; use one process per container instead
• Sticky sessions or external store required for cluster mode state sharing
• Set environment file permissions to 600 with root:nodeapp ownership

On-Call Runbook Starters

• High CPU: sync JSON.parse on huge payloads, missing await, inefficient middleware
• NODE_OPTIONS max-old-space-size should be 75% of container memory limit
• Container apps must bind 0.0.0.0:PORT to be routable through ingress
• Empty kubectl endpoints means all pods not ready; common 502 root cause
• Ingress proxy-read-timeout must exceed app p99 upstream processing time
• Compare RPS against baseline when CPU spikes; use kubectl top pods to find hot pods

Runtime Ops Best Practices

• Document and implement SIGTERM graceful shutdown on every HTTP service
• Fail /ready during drain; keep /health cheap
• Exit on uncaughtException for APIs unless runbook says otherwise
• Set --max-old-space-size below container memory limit
• Structured JSON logs with requestId to stdout
• Runbooks for high CPU, memory, and 502 incidents