The Node.js Resilience Model
Resilience is the deliberate practice of bounding how far a failure can spread through a system, rather than assuming - or hoping - that every downstream dependency will always respond correctly and on time.
//
Search across all documentation pages
Resilience is the deliberate practice of bounding how far a failure can spread through a system, rather than assuming - or hoping - that every downstream dependency will always respond correctly and on time.
The resilience section covers a specific set of patterns: timeouts, retries with backoff, circuit breakers, and graceful degradation.
Read individually, they can look like an arbitrary toolbox; read as one model, they're actually a sequence of questions a request asks about a struggling dependency, each pattern answering the question the one before it couldn't.
Resilience Basics shows working code for each pattern; this page is the reasoning that connects them and explains why the order they're applied in matters as much as which ones you choose.
A failure domain is the boundary within which a failure is contained - the set of things that break together when one thing breaks.
Without deliberate design, a Node backend's failure domains are much larger than they need to be: a single slow database query can exhaust your connection pool, which stalls every other request using that pool, which cascades into your service failing entirely - one dependency's bad day becomes your entire process's bad day.
Resilience patterns exist to shrink failure domains back down to the size of the actual problem.
A useful analogy is a ship's watertight bulkheads: a hull breach in one compartment floods that compartment, and the compartment next to it, but the doors between sections are built specifically so the ship as a whole stays afloat instead of one puncture sinking it entirely.
Each resilience pattern in this section is a different kind of bulkhead, answering a different question in what's actually a fixed sequence:
Skipping an earlier layer doesn't just remove that protection - it undermines the layers after it: retries without a timeout can retry a call that's still hanging from the first attempt, and a circuit breaker without a bulkhead can still be starved of capacity by concurrent calls piling up before it ever trips.
The sequence above isn't just a list - it's a control flow, and seeing it as one diagram clarifies why each layer exists:
request
│
▼
┌─────────────┐ exceeds limit ┌──────────────┐
│ timeout │ ─────────────────▶ │ fail/retry │
└─────────────┘ └──────┬───────┘
│ retries exhausted, or
│ error rate crosses threshold
▼
┌──────────────┐
│ circuit │ open: fail fast,
│ breaker │ no calls attempted
└──────┬───────┘
│ circuit open, or
│ dependency non-critical
▼
┌──────────────┐
│ graceful │
│ degradation │ serve a reduced
└──────────────┘ but working response
A circuit breaker's internal state machine is the mechanism worth pinning down precisely, because its middle state is the part people usually get wrong.
type BreakerState = "closed" | "open" | "half-open";
// closed: calls pass through normally; failures are counted
// open: calls fail immediately, no request reaches the dependency
// half-open: exactly one trial call is allowed through, to test recovery
// - success -> closed again; failure -> open again, cooldown resetsClosed is the normal state - calls go through, and failures are just counted toward a threshold.
Open is the contained-failure state - the breaker fails every call immediately, without even attempting the network request, which is what actually protects your service's own capacity (threads, sockets, event-loop time) from being consumed by calls to something that's already known to be down.
Half-open exists because a breaker that just flips back to closed after a fixed cooldown risks slamming a barely-recovering dependency with your full traffic all at once; letting exactly one (or a small, controlled number of) trial calls through first answers "has it actually recovered?" before committing the rest of your traffic back to it.
This is also where retries and circuit breakers interact in a way that's easy to get backwards: a retry loop that keeps retrying against an open circuit is pointless work, since the breaker is already failing fast by design - retry logic should check breaker state, not retry blindly underneath it.
Backoff - waiting progressively longer between retry attempts, usually with random jitter added - exists to prevent every failed client from retrying at the exact same moment a struggling dependency is trying to recover, which would otherwise turn a brief blip into a synchronized wave of renewed load (a retry storm) right when the dependency can least handle it.
At scale, the biggest failure mode resilience patterns guard against isn't a single dependency going down - it's cascading failure, where a slowdown in one service propagates through synchronous call chains until an unrelated part of the system fails too, purely because it was waiting on something waiting on something else.
Bulkheads (bounding concurrent calls to a specific dependency, via a semaphore, a dedicated connection pool, or a worker limit) exist specifically to stop this propagation at the point of contact: a slow vendor can exhaust its own allotted capacity without touching the capacity reserved for everything else your service does.
Chaos engineering - deliberately injecting failures (killed instances, added latency, dropped connections) into a system to observe whether its resilience patterns actually behave as designed - exists because these patterns are notoriously hard to verify by reading code alone; a circuit breaker's threshold, a timeout's duration, and a retry's backoff curve all interact in ways that are much easier to get wrong on paper than to observe under an injected failure.
Observability is not optional here: each resilience layer needs its own signal to be useful operationally - a rising timeout rate, a circuit breaker's open/half-open transitions, a growing retry count - because without them, a resilience pattern silently doing its job (containing a failure) can look identical from the outside to a resilience pattern silently failing to help at all.
Where this logic actually lives has also shifted over the industry's history: early Node services hand-rolled retry loops inline; libraries like opossum and cockatiel standardized circuit-breaker and retry logic as reusable, testable components; and increasingly, service meshes (Envoy, Istio) implement timeouts, retries, and circuit breaking at the infrastructure layer, outside application code entirely, which trades application-level flexibility for consistency enforced across every service regardless of language.
| Where Resilience Logic Lives | Strength | Weakness | Best Fit |
|---|---|---|---|
| Inline application code | Full control; no new infrastructure dependency | Easy to get subtly wrong; inconsistent across services/teams | Small services, or logic needing business-specific nuance |
| Library (opossum, cockatiel) | Tested, reusable, documented failure semantics | Still configured and owned per-service; can drift between services | Most Node services - the default choice |
| Service mesh / sidecar | Consistent policy across every service, language-agnostic | Real infrastructure to run and operate; less fine-grained control per call | Large multi-service, polyglot architectures |
The deliberate practice of bounding how far a failure spreads through your system - via timeouts, retries, circuit breakers, bulkheads, and graceful degradation - rather than assuming every dependency will always respond correctly and on time.
Because each one answers a question the previous one leaves open: a timeout bounds how long you wait, a retry decides whether to try again, a circuit breaker decides when to stop trying at all, and graceful degradation decides what the user sees when nothing worked. Skipping an earlier layer undermines the ones after it - retrying without a timeout, for instance, can retry a call that's still hanging.
After a cooldown period in the open state, the breaker allows a small, controlled number of trial calls through instead of resuming full traffic immediately. If those trial calls succeed, the breaker closes and normal traffic resumes; if they fail, it reopens and the cooldown restarts - this avoids slamming a barely-recovering dependency with full load all at once.
Because you can't tell, from a timeout or a dropped connection, whether the original request actually succeeded on the server before the response was lost - retrying it might execute the same operation (like a payment charge) a second time. Retries are only safe when the operation is idempotent, or is protected by an explicit idempotency key the server can deduplicate on.
A circuit breaker decides whether to attempt a call at all, based on that dependency's recent failure history. A bulkhead limits how many calls can be in flight at once to a given dependency, regardless of whether they're succeeding - so a bulkhead protects your own capacity even from a dependency that's merely slow rather than fully failing.
Without jitter, every client that failed at the same moment retries at the same moment again, creating a synchronized wave of load (a retry storm) exactly when a recovering dependency can least handle it. Random jitter spreads those retries out in time, smoothing the load instead of concentrating it.
Yes - resilience always trades some added latency and code complexity for containment, so a call to a genuinely reliable, low-risk internal dependency with tight coupling to its caller may not need the full sequence. The judgment call is about which failure modes are plausible and costly enough to be worth guarding against, not applying every pattern uniformly everywhere.
A single dependency failure is contained if resilience patterns are working; a cascading failure is what happens when that containment doesn't hold, and a slowdown in one service propagates through synchronous call chains until unrelated parts of the system fail too, purely from waiting on something that's waiting on something else.
Because a resilience layer working correctly (quietly containing a failure) and one silently failing to help can look identical from the outside without dedicated signals - a timeout rate, a circuit breaker's state transitions, a retry count - to tell them apart. Without those signals, you can't verify the patterns are actually doing their job.
A service mesh (like Envoy or Istio) enforces consistent timeout, retry, and circuit-breaking policy across every service regardless of language, at the cost of running and operating real infrastructure and losing some of the fine-grained, business-specific control that inline application logic allows.
No - it's a verification technique, not a design one. Deliberately injecting failures (killed instances, added latency) tests whether your timeout durations, retry policies, and breaker thresholds actually behave as intended together, but you still have to reason through the sequence and interactions to design them in the first place.
Done well: a product page falls back to a cached recommendation list, with a visible note, when the live recommendation service is down. Done badly: the same page silently omits recommendations with no indication anything is wrong, leaving both users and on-call engineers unaware a dependency has failed at all.
Stack versions: This page was written for Node.js 24 LTS and TypeScript 5.6+.
Reviewed by Chris St. John·Last updated Jul 15, 2026