Resilience Best Practices

Document retry safety per endpoint in OpenAPI. These rules keep Node.js services available when dependencies slow down or fail.

How to Use This List

• Apply to every new outbound integration and HTTP route.
• Pair with load tests that simulate downstream 503 latency.
• Add OpenAPI x-retry-safe and x-idempotency extensions per operation.
• Review during incident postmortems when retries amplified outages.

A - Timeouts and Deadlines

• Every fetch and HTTP client call uses AbortSignal.timeout. No infinite waits.
• DB statement_timeout or driver query timeout set. Server-side kill for runaway SQL.
• Outbound timeout < handler deadline < server requestTimeout < LB idle. Staggered budget.
• Connection acquire timeout on pools. Log pool wait metrics.
• BullMQ jobs have timeout option. Workers do not run unbounded.

B - Retries and Idempotency

• Retry 5xx and network errors only, not 4xx. Classify retryable vs fatal.
• Exponential backoff with jitter between attempts. Max 3-5 attempts on sync paths.
• Idempotency-Key on POST that creates charges or orders. Document in OpenAPI.
• Store idempotency key -> response mapping 24h+ for payments. Return cached on duplicate.
• Stop retrying when circuit is open. Fail fast locally.

C - Circuit Breakers and Degradation

• opossum (or equivalent) per downstream dependency. Separate breakers per service.
• Fallback only on tier 1/2 features. Never fake payment or auth success.
• Return meta.degraded in API responses when shedding features. Client can adjust UI.
• Feature flags to disable optional paths during incidents. FEATURE_X=off without redeploy.
• Alert on circuit open events. Page when core dependency breaker opens.

D - Health and Lifecycle

• Liveness cheap; readiness checks DB and critical deps. No slow checks on liveness.
• Graceful SIGTERM: server.close, drain pools, sdk.shutdown(). Match k8s grace period.
• Fail readiness during migrations if needed. Traffic shifts to old pods.
• Queue backlog monitored as leading indicator. Depth alert before worker timeouts.
• Chaos or fault injection in staging quarterly. Validate degradation paths work.

E - Documentation and Contracts

• OpenAPI documents idempotency and retry-safe per operation. Consumers know safe retry rules.
• Runbook per dependency: timeout, breaker thresholds, fallback behavior. On-call ready.
• Classify dependencies tier 0/1/2 in service README. Clarifies degradation policy.
• SLI includes dependency timeout rate. Not only 5xx from your service.
• Post-incident review asks "did retries amplify?" Tune breaker and retry caps.

FAQs

Related

• Resilience Basics - introductory examples
• Timeouts Everywhere - timeout matrix
• Retries with Backoff - idempotency patterns
• OpenAPI & Swagger - document retry extensions

Stack versions: This page was written for Node.js 24.18.0 (Active LTS), npm 10+, TypeScript 5.6+, Express 5, Fastify 5, and NestJS 11.