Streams Best Practices

Q: What about S3 upload?

@aws-sdk/lib-storage Upload with stream body - respects backpressure to S3 multipart API.

Q: How to test backpressure?

Writable with slow async _write and producer pushing fast - assert memory stable over time.

Streams keep Node services memory-stable - these rules prevent hung sockets, OOM from unbounded buffers, and silent data loss on unhandled error events.

How to Use This List

Apply to every file upload, download, ETL, and gzip path.
Treat manual .pipe().pipe() without error handlers as a review blocker.
Load test streaming endpoints with slow clients, not just fast localhost curls.
Pair with Buffers Best Practices for size caps.

A - Wiring and Errors

Use stream/promises.pipeline instead of bare pipe chains. Automatic destroy on error.
Wrap await pipeline(...) in try/catch in HTTP handlers. Map to 500 only if headers not sent.
Attach error listeners if not using pipeline. Rare exceptions for legacy code with migration ticket.
Destroy upstream reads when HTTP client aborts. Listen req.on('aborted') or AbortSignal.
Implement _flush on Transforms buffering partial frames/lines. Do not lose trailing data.

B - Memory and Backpressure

Stream any payload you would not comfortably fit in one Buffer. Rule of thumb: multi-MB+.
Respect write() returning false in custom loops. Pause until drain.
Tune highWaterMark with measurements, not guesses. Watch heap and p99 latency.
Avoid unbounded Array of chunks before concat. Use pipeline or bounded queue.
Cap object mode stream objects at reasonable size. Not a substitute for byte limits on huge JSON.

C - HTTP Streaming

Set content-type and disposition headers before body bytes. Required for downloads and NDJSON.
Disable proxy buffering for streaming routes in nginx/CDN config. Or use presigned URLs for huge files.
Align load balancer idle timeouts with longest expected stream. Or send periodic keepalive bytes where protocol allows.
Do not double-compress gzipped assets. Bypass compression middleware on precompressed paths.
Use Content-Length when size known. Better CDN behavior and download progress.

D - Formats and Transforms

Keep gzip/crypto transforms in pipeline order documented. Bytes before object mode conversions.
Parse binary frames with incremental Transform, not whole-buffer concat. See Binary Protocols.
Validate max frame/body size before allocation. Reject malicious length prefixes.
Prefer Readable.from async iterables for simple sources. Subclass Readable when stateful _read needed.
Bridge Web Streams with fromWeb/toWeb at fetch boundaries. Do not buffer response.arrayBuffer() by default.

E - Operations

Log stream errors with route name and correlation ID, not chunk contents. Avoid PII in logs.
Monitor open file descriptors during export load tests. Leaks show up as EMFILE.
Integration-test chunked responses with slow consumer client. Supertest may hide streaming bugs.
Document which routes stream in runbooks. Incident responders check proxy buffering first.
Prefer presigned object storage URLs when API gateway bandwidth is costly. Stream through API only when audit required.

FAQs

Why is pipeline mandatory?

Manual pipe chains often miss error propagation - production leaks sockets and file handles.

When is Buffer OK instead of streams?

Small known-size payloads (KB scale), crypto HMAC over bounded body, in-memory transforms in tests.

Does Express body parser stream?

Default JSON parser buffers - use raw stream or busboy for multipart streaming uploads.

Fastify and streams?

Fastify supports stream responses natively - still use pipeline for multi-stage processing before reply.

What about S3 upload?

@aws-sdk/lib-storage Upload with stream body - respects backpressure to S3 multipart API.

Can I pipeline into Redis?

Redis protocol is request/response - usually not a Writable stream sink; use commands or dedicated client APIs.

How to test backpressure?

Writable with slow async _write and producer pushing fast - assert memory stable over time.

Are object mode streams default?

No - opt in explicitly when streaming records/events, not bytes.

Windows pipe quirks?

Same API - test large file streams on target deploy OS for antivirus interference.

Relation to worker_threads?

CPU-heavy per-chunk work may belong in worker - stream coordinates flow on main thread.

Does NestJS stream files?

Use underlying Express/Fastify adapter patterns or @nestjs/platform-fastify stream reply APIs.

Top mistake?

Reading entire upload into Buffer on edge - always stream to storage with size limits.

stream/promises.pipeline - API reference
Backpressure - pause/drain
Streaming HTTP Responses - proxies
Buffers Best Practices - size limits

Node.js API Documentation - Complete reference for core modules and APIs.
Node.js Best Practices (GitHub) - Curated production best practices.
DigitalOcean Node.js Tutorials - Production-oriented how-tos.
Node.js Design Patterns, 4th Edition - Production-grade application patterns.
Node.js Cookbook, 5th Edition - Practical server-side recipes.

Stack versions: This page was written for Node.js 24.18.0 (Active LTS), npm 10+, TypeScript 5.6+, Express 5, Fastify 5, and NestJS 11.

Streams Best Practices

Streams keep Node services memory-stable - these rules prevent hung sockets, OOM from unbounded buffers, and silent data loss on unhandled error events.

How to Use This List

Apply to every file upload, download, ETL, and gzip path.
Treat manual .pipe().pipe() without error handlers as a review blocker.
Load test streaming endpoints with slow clients, not just fast localhost curls.
Pair with Buffers Best Practices for size caps.

A - Wiring and Errors

Use stream/promises.pipeline instead of bare pipe chains. Automatic destroy on error.
Wrap await pipeline(...) in try/catch in HTTP handlers. Map to 500 only if headers not sent.
Attach error listeners if not using pipeline. Rare exceptions for legacy code with migration ticket.
Destroy upstream reads when HTTP client aborts. Listen req.on('aborted') or AbortSignal.
Implement _flush on Transforms buffering partial frames/lines. Do not lose trailing data.

B - Memory and Backpressure

Stream any payload you would not comfortably fit in one Buffer. Rule of thumb: multi-MB+.
Respect write() returning false in custom loops. Pause until drain.
Tune highWaterMark with measurements, not guesses. Watch heap and p99 latency.
Avoid unbounded Array of chunks before concat. Use pipeline or bounded queue.
Cap object mode stream objects at reasonable size. Not a substitute for byte limits on huge JSON.

C - HTTP Streaming

Set content-type and disposition headers before body bytes. Required for downloads and NDJSON.
Disable proxy buffering for streaming routes in nginx/CDN config. Or use presigned URLs for huge files.
Align load balancer idle timeouts with longest expected stream. Or send periodic keepalive bytes where protocol allows.
Do not double-compress gzipped assets. Bypass compression middleware on precompressed paths.
Use Content-Length when size known. Better CDN behavior and download progress.

D - Formats and Transforms

Keep gzip/crypto transforms in pipeline order documented. Bytes before object mode conversions.
Parse binary frames with incremental Transform, not whole-buffer concat. See Binary Protocols.
Validate max frame/body size before allocation. Reject malicious length prefixes.
Prefer Readable.from async iterables for simple sources. Subclass Readable when stateful _read needed.
Bridge Web Streams with fromWeb/toWeb at fetch boundaries. Do not buffer response.arrayBuffer() by default.

E - Operations

Log stream errors with route name and correlation ID, not chunk contents. Avoid PII in logs.
Monitor open file descriptors during export load tests. Leaks show up as EMFILE.
Integration-test chunked responses with slow consumer client. Supertest may hide streaming bugs.
Document which routes stream in runbooks. Incident responders check proxy buffering first.
Prefer presigned object storage URLs when API gateway bandwidth is costly. Stream through API only when audit required.

FAQs

Why is pipeline mandatory?

Manual pipe chains often miss error propagation - production leaks sockets and file handles.

When is Buffer OK instead of streams?

Small known-size payloads (KB scale), crypto HMAC over bounded body, in-memory transforms in tests.

Does Express body parser stream?

Default JSON parser buffers - use raw stream or busboy for multipart streaming uploads.

Fastify and streams?

Fastify supports stream responses natively - still use pipeline for multi-stage processing before reply.

What about S3 upload?

@aws-sdk/lib-storage Upload with stream body - respects backpressure to S3 multipart API.

Can I pipeline into Redis?

Redis protocol is request/response - usually not a Writable stream sink; use commands or dedicated client APIs.

How to test backpressure?

Writable with slow async _write and producer pushing fast - assert memory stable over time.

Are object mode streams default?

No - opt in explicitly when streaming records/events, not bytes.

Windows pipe quirks?

Same API - test large file streams on target deploy OS for antivirus interference.

Relation to worker_threads?

CPU-heavy per-chunk work may belong in worker - stream coordinates flow on main thread.

Does NestJS stream files?

Use underlying Express/Fastify adapter patterns or @nestjs/platform-fastify stream reply APIs.

Top mistake?

Reading entire upload into Buffer on edge - always stream to storage with size limits.

stream/promises.pipeline - API reference
Backpressure - pause/drain
Streaming HTTP Responses - proxies
Buffers Best Practices - size limits

Node.js API Documentation - Complete reference for core modules and APIs.
Node.js Best Practices (GitHub) - Curated production best practices.
DigitalOcean Node.js Tutorials - Production-oriented how-tos.
Node.js Design Patterns, 4th Edition - Production-grade application patterns.
Node.js Cookbook, 5th Edition - Practical server-side recipes.

Stack versions: This page was written for Node.js 24.18.0 (Active LTS), npm 10+, TypeScript 5.6+, Express 5, Fastify 5, and NestJS 11.

Streams Best Practices

Search Documentation

Streams Best Practices