BullMQ Failed Jobs: Monitor and Retry in Production

BullMQ jobs fail silently by default: a failed job simply moves to the failed set while your app keeps returning success to users, so production needs per-job retries with backoff, handlers for failed and stalled events, a dead-letter pattern, and an alert when the failed count rises.

Background jobs send emails, charge cards, and sync data. When they break, the user-visible request already succeeded, so nobody notices until a customer asks where their invoice went.

Step 1: Configure Retries and Backoff on Every Job

import { Queue } from 'bullmq'

export const emailQueue = new Queue('email', { connection })

await emailQueue.add('welcome', { userId }, {
  attempts: 5,
  backoff: { type: 'exponential', delay: 2000 },
  removeOnComplete: { age: 3600, count: 1000 },
  removeOnFail: { age: 7 * 24 * 3600 }, // keep failures a week for debugging
})

Without removeOnComplete and removeOnFail, finished jobs accumulate in Redis and eat memory. Without attempts, a single transient error is permanent.

Step 2: Make Workers Idempotent

A job can run more than once: retries, stalled recovery, or a worker crash after the side effect but before acknowledgement. Use an idempotency key for anything with an external effect:

const worker = new Worker('email', async job => {
  const key = `sent:${job.name}:${job.data.userId}`
  if (await redis.set(key, '1', 'EX', 86400, 'NX') === null) return // already done
  await sendEmail(job.data)
}, { connection, concurrency: 5 })

This is the same principle that protects payments, covered in Stripe webhook failures.

Step 3: Listen for Failures and Stalls

worker.on('failed', (job, err) => {
  const final = job && job.attemptsMade >= (job.opts.attempts ?? 1)
  logger[final ? 'error' : 'warn']({ jobId: job?.id, name: job?.name, err }, 'job failed')
  if (final) alertTeam(`Job ${job?.name} exhausted retries`, err)
})

worker.on('stalled', jobId => logger.warn({ jobId }, 'job stalled'))
worker.on('error', err => logger.error({ err }, 'worker error')) // unhandled = crash

A stalled job means a worker stopped renewing its lock, usually because the event loop was blocked or the process died. If you see stalls, check event loop blocking and memory kills (Docker exit 137). Attach the worker.on('error') handler; without it, an error event can crash the process, as described in Node.js production error handling.

Step 4: Dead-Letter Pattern

Once retries are exhausted, move the job to a separate queue where a human or a repair script can inspect it:

const deadQueue = new Queue('email-dead', { connection })

worker.on('failed', async (job, err) => {
  if (job && job.attemptsMade >= (job.opts.attempts ?? 1)) {
    await deadQueue.add('dead', { original: job.data, error: err.message, name: job.name })
  }
})

Step 5: Alert on Queue Health, Not Only Errors

const counts = await emailQueue.getJobCounts('waiting', 'active', 'failed', 'delayed')

Export these counts every minute to your monitoring. A dashboard such as Bull Board helps for manual inspection.

Redis Settings That Matter

BullMQ expects Redis to never evict its keys: set maxmemory-policy noeviction, otherwise jobs can vanish under memory pressure. On managed or serverless Redis, check connection limits and timeouts; see Redis Upstash errors on serverless. Run workers as long-lived processes (Render, Railway, a container), not inside short-lived serverless functions; see Render and Railway monitoring.

Common Failure Causes

• Job payload too large or not JSON serializable.
• Worker deploys while jobs are active; use graceful shutdown with await worker.close().
• A third-party API rate limit; see OpenAI API 429 handling for backoff patterns.
• No concurrency cap, overwhelming a database (connection pool exhaustion).

Let Relia Watch the Part Users Cannot See

Relia watches production behavior, including failures behind the request, and when job errors pile up it writes the incident up in plain words: what happened, why it thinks so (with a confidence level), and one thing to do next. On request it writes a fix plan with a "Copy for your AI assistant" button; Relia doesn't change code itself, and after you ship it checks the fix against live traffic.

FAQ

Why are my BullMQ jobs stuck in "waiting"?

No worker is running, workers are paused, or concurrency is saturated by long jobs. Check worker logs and process health first.

What is a stalled job in BullMQ?

The worker holding the job stopped renewing its lock, so BullMQ reassigns it. It usually means a blocked event loop or a crashed process.

How many retries should I use?

Three to five attempts with exponential backoff covers most transient failures. Alert and dead-letter after that.