Queues Best Practices
Monitor queue depth; alert before consumers drown. Queues decouple HTTP from slow work only when ops discipline matches.
How to Use This List
- Dashboard waiting/active/failed counts per queue before launch
- Load test producer burst with fixed worker count
- Pair with Idempotency Keys for payment paths
- Run quarterly DLQ redrive drill
A - Producer (API) Rules
- HTTP returns 202 with job id for work >500ms expected duration.
- Enqueue is fast - no heavy serialization or DB work in enqueue path.
- Stable jobId / idempotency key on client-retried mutations.
- Correlation id in job payload for tracing across API and worker logs.
- Separate queues by domain and blast radius (
email,billing,media).
B - Worker Rules
- Workers idempotent - safe if job runs twice (stalled SQS/BullMQ delivery).
- Concurrency tuned per job type - I/O vs CPU bound profiles differ.
- Throw on transient errors to trigger retry; catch permanent errors to fail fast.
- Side effects after local idempotency check and DB commit.
- Progress updates for jobs >30s user-visible wait.
C - Retries and DLQ
- Exponential backoff on outbound vendor calls inside jobs.
- Max attempts capped (3-10 depending on cost).
- DLQ configured (SQS redrive or Bull failed set) with alarm.
- Runbook: inspect DLQ, fix, redrive documented in on-call wiki.
- Poison message identification - log job name + payload hash, not secrets.
D - Operations
- Metrics: depth, age of oldest message, processing rate, fail rate.
- Autoscale workers on depth (k8s HPA custom metric or ECS step scaling).
- Graceful shutdown -
worker.close()with grace period > p99 job time. - Redis/ElastiCache memory monitored when using BullMQ.
- Visibility timeout / lockDuration > p99 handler duration.
E - Payload and Security
- Payload < 64KB ideally; S3 reference for large files.
- No raw PII unless necessary - pass ids and load in worker.
- Worker IAM least privilege for S3, SQS, Stripe, etc.
- Encrypt sensitive job fields if queue is multi-tenant admin accessible.
- Admin retry UI (Bull Board) behind auth and audit log.
F - Scheduling
- No duplicate cron across replicas without lock or k8s CronJob.
- BullMQ repeatable jobs use stable jobId.
- Missed run policy documented (catch-up vs skip).
- Heavy schedules enqueue worker jobs instead of running inline in API.
- UTC timezone explicit in cron definitions.
FAQs
How deep is too deep?
Sustained growth over 15 minutes is incident. Steady state near zero with spikes OK.
One worker process or many?
Many pods with moderate concurrency beats one monster pod for fault tolerance.
Sync fallback if queue down?
Rare. Return 503 and alert. Do not block HTTP 30s on Redis outage.
BullMQ vs SQS?
BullMQ if Redis exists. SQS for AWS-managed with less Redis ops.
Priority queues?
Use separate queues for premium tier instead of starving bulk work.
Job ordering required?
SQS FIFO or single concurrency per MessageGroupId. BullMQ single consumer queue.
Testing retries?
Mock vendor 503; assert attempts count and final DLQ placement.
NestJS same rules?
Yes. Processors must be idempotent; Bull Board still recommended.
Biggest outage pattern?
Queue depth silent growth until Redis OOM or SLA miss. Alert early.
Job versioning?
Include schemaVersion in payload; workers handle N and N-1 during deploy.
Related
- Queues Basics - fundamentals
- BullMQ - Redis queues
- AWS SQS Consumers - managed queues
- Idempotency Keys - duplicate safety
- Graceful Shutdown - worker drain
Stack versions: This page was written for Node.js 24.18.0 (Active LTS), npm 10+, TypeScript 5.6+, Express 5, Fastify 5, and NestJS 11.