Observability Best Practices

Alert on symptoms, not every stack trace. These rules keep Node.js observability actionable, affordable, and correlated across logs, metrics, and traces.

How to Use This List

• Apply when standing up a new service or migrating APM vendors.
• Pair with SLOs & Error Budgets for paging policy.
• Review cardinality before exporting new custom metrics.
• Run game days to verify traces cover critical user journeys.

A - Metrics and SLOs

• Export RED metrics for every HTTP service. Rate, errors (5xx), duration histogram.
• Add pool wait and queue depth where applicable. Leading indicators before 5xx.
• Define SLO with error budget policy in writing. Not dashboard-only.
• Page on multi-window SLO burn, not single log line. Fast and slow burn rules.
• Normalize route labels (/users/:id). Never UUIDs in metric dimensions.

B - Traces and Instrumentation

• Load instrumentation.ts before framework imports. HTTP auto-instrumentation requirement.
• One instrumentation path per process. OTel SDK XOR vendor agent, not both.
• Head-sample traces in production (1-10%). Tail-sample errors if vendor supports.
• Manual spans on critical business operations. Payment, auth, enqueue.
• Propagate W3C traceparent on outbound HTTP. Downstream services continue trace.

C - Logs and Correlation

• Inject trace_id into Pino logs via mixin or ALS. Log-trace correlation in APM.
• Keep requestId alongside trace_id during migration. Support and legacy tools.
• Structured JSON only; no PII in span attributes or log fields. IDs not emails.
• Separate dev/staging/prod telemetry backends. No prod trace leakage to dev accounts.
• info for lifecycle; error for failures needing action. Same rules as logging section.

D - Dashboards and Alerts

• One golden dashboard per service. RED + pool + queue + loop delay.
• Runbooks linked from alert annotations. Not embedded in every span.
• Alert on symptom thresholds (p95, error rate). Not CPU alone.
• Review alert fatigue quarterly. Delete alerts that never led to action.
• Health check excluded from SLO or logged at debug. Probe traffic skews metrics.

E - Operations and Cost

• Graceful sdk.shutdown() on SIGTERM. Flush spans before pod kill.
• OTel Collector for vendor export in k8s. App stays vendor-neutral OTLP.
• Document OTEL_SERVICE_NAME per deployment. Service catalog accuracy.
• Monitor telemetry ingest cost monthly. Sampling adjustments when span volume spikes.
• Validate observability in load tests before launch. Metrics and traces under expected RPS.

FAQs

Minimum viable observability?

Pino JSON logs with requestId + RED metrics + 10% trace sampling on staging before prod.

Logs vs traces priority?

Logs first (cheap, always on). Metrics second (SLOs). Traces third (debugging depth).

Who owns the collector?

Platform/SRE team typically. App teams own instrumentation.ts and service.name.

Missing traces on one route?

Check instrumentation load order and whether route bypasses instrumented HTTP stack.

GraphQL observability?

HTTP-level RED plus optional per-operation spans with bounded operation name cardinality.

Workers and cron?

Separate service.name. Same OTel bootstrap pattern; trace job execution as root span.

Security scanning of telemetry?

Treat traces/logs as sensitive data stores - RBAC, retention TTL, no secrets in attributes.

Relation to logging best practices?

Logging Best Practices covers log specifics; this list covers full observability stack.

Related

• Observability Basics - three pillars intro
• OpenTelemetry Node SDK - SDK configuration
• Metrics that Matter - what to export
• Logging Best Practices - structured logging rules

Stack versions: This page was written for Node.js 24.18.0 (Active LTS), npm 10+, TypeScript 5.6+, Express 5, Fastify 5, and NestJS 11.