Observability Best Practices
Alert on symptoms, not every stack trace. These rules keep Node.js observability actionable, affordable, and correlated across logs, metrics, and traces.
How to Use This List
- Apply when standing up a new service or migrating APM vendors.
- Pair with SLOs & Error Budgets for paging policy.
- Review cardinality before exporting new custom metrics.
- Run game days to verify traces cover critical user journeys.
A - Metrics and SLOs
- Export RED metrics for every HTTP service. Rate, errors (5xx), duration histogram.
- Add pool wait and queue depth where applicable. Leading indicators before 5xx.
- Define SLO with error budget policy in writing. Not dashboard-only.
- Page on multi-window SLO burn, not single log line. Fast and slow burn rules.
- Normalize route labels (
/users/:id). Never UUIDs in metric dimensions.
B - Traces and Instrumentation
- Load
instrumentation.tsbefore framework imports. HTTP auto-instrumentation requirement. - One instrumentation path per process. OTel SDK XOR vendor agent, not both.
- Head-sample traces in production (1-10%). Tail-sample errors if vendor supports.
- Manual spans on critical business operations. Payment, auth, enqueue.
- Propagate W3C
traceparenton outbound HTTP. Downstream services continue trace.
C - Logs and Correlation
- Inject
trace_idinto Pino logs via mixin or ALS. Log-trace correlation in APM. - Keep
requestIdalongsidetrace_idduring migration. Support and legacy tools. - Structured JSON only; no PII in span attributes or log fields. IDs not emails.
- Separate dev/staging/prod telemetry backends. No prod trace leakage to dev accounts.
- info for lifecycle; error for failures needing action. Same rules as logging section.
D - Dashboards and Alerts
- One golden dashboard per service. RED + pool + queue + loop delay.
- Runbooks linked from alert annotations. Not embedded in every span.
- Alert on symptom thresholds (p95, error rate). Not CPU alone.
- Review alert fatigue quarterly. Delete alerts that never led to action.
- Health check excluded from SLO or logged at debug. Probe traffic skews metrics.
E - Operations and Cost
- Graceful
sdk.shutdown()on SIGTERM. Flush spans before pod kill. - OTel Collector for vendor export in k8s. App stays vendor-neutral OTLP.
- Document
OTEL_SERVICE_NAMEper deployment. Service catalog accuracy. - Monitor telemetry ingest cost monthly. Sampling adjustments when span volume spikes.
- Validate observability in load tests before launch. Metrics and traces under expected RPS.
FAQs
Minimum viable observability?
Pino JSON logs with requestId + RED metrics + 10% trace sampling on staging before prod.
Logs vs traces priority?
Logs first (cheap, always on). Metrics second (SLOs). Traces third (debugging depth).
Who owns the collector?
Platform/SRE team typically. App teams own instrumentation.ts and service.name.
Missing traces on one route?
Check instrumentation load order and whether route bypasses instrumented HTTP stack.
GraphQL observability?
HTTP-level RED plus optional per-operation spans with bounded operation name cardinality.
Workers and cron?
Separate service.name. Same OTel bootstrap pattern; trace job execution as root span.
Security scanning of telemetry?
Treat traces/logs as sensitive data stores - RBAC, retention TTL, no secrets in attributes.
Relation to logging best practices?
Logging Best Practices covers log specifics; this list covers full observability stack.
Related
- Observability Basics - three pillars intro
- OpenTelemetry Node SDK - SDK configuration
- Metrics that Matter - what to export
- Logging Best Practices - structured logging rules
Stack versions: This page was written for Node.js 24.18.0 (Active LTS), npm 10+, TypeScript 5.6+, Express 5, Fastify 5, and NestJS 11.