Background Jobs and Queues: Stop Making Users Wait

A user clicks "Sign up," and your server creates the account, sends a welcome email, resizes their avatar, notifies the CRM and logs an analytics event, all before responding. If the email provider is slow that day, so is your signup. If it's down, signups fail.

Most of that work doesn't need to happen while the user waits. That's what background jobs are for.

The idea

Instead of doing slow work inside the request:

1. The request handler saves what's essential and adds a job to a queue: "send welcome email to user 42."
2. It responds to the user immediately.
3. A separate worker process picks up jobs from the queue and runs them.

Request  ->  save user  ->  enqueue "welcome_email"  ->  201 Created (fast)
                                     |
                              Worker picks it up  ->  sends email

What belongs in a background job

• Sending emails and notifications (email deliverability)
• Image and video processing (file uploads)
• Generating reports, exports and PDFs
• Calling slow or unreliable third-party APIs
• Processing incoming webhooks (webhooks done right)
• Bulk data updates and imports
• Anything that can be retried if it fails

Rule of thumb: if the user doesn't need the result to see the next screen, it's a candidate.

Common tools

Every major ecosystem has a mature queue library:

Stack	Popular options
Node.js	BullMQ (Redis-backed)
Python	Celery, RQ, Dramatiq
Ruby	Sidekiq
PHP / Laravel	Laravel Queues
Postgres-based	Libraries that use your existing database as the queue

Using your existing Postgres database as the queue is a great way to start: no extra infrastructure. Move to Redis or a dedicated broker when volume demands it.

Retries and backoff

Jobs fail: networks blip, APIs return 503, rate limits hit. A good queue retries automatically, with exponential backoff (wait 10 seconds, then 30, then 2 minutes...) so you don't hammer a struggling service.

After a maximum number of attempts, move the job to a dead-letter queue (or "failed jobs" list) for a human to inspect. Don't retry forever, and don't silently drop it.

Make jobs idempotent

Retries mean a job might run more than once: it may have succeeded but crashed before reporting success. Design jobs so running twice is harmless:

• Check before acting: "has this email already been sent for this order?"
• Use unique IDs and database constraints to prevent duplicates.
• Pass idempotency keys to APIs that support them, like payment providers.

async function sendReceipt({ orderId }) {
  const order = await db.orders.find(orderId)
  if (order.receiptSentAt) return          // already done, safe to exit
  await email.send(receiptFor(order))
  await db.orders.update(orderId, { receiptSentAt: new Date() })
}

Pass IDs, not whole objects

Enqueue { orderId: 991 }, not the entire order. The job loads fresh data when it runs, so it never works with stale information, and queue payloads stay small.

Scheduled jobs (cron)

Some work runs on a schedule rather than in response to a request: nightly reports, cleaning up expired sessions, sending reminders.

• Most queue libraries support repeatable or scheduled jobs.
• Plain cron on a server works too.
• GitHub Actions can run simple scheduled tasks. (Scheduled workflows.)

Schedule in UTC, and make sure a job can't overlap with its previous run if it takes longer than expected. (Time zone pitfalls.)

Serverless options

On serverless platforms there's no always-on worker process. Options include your platform's queue or background function features, managed queue services that call your endpoints, and cron triggers. The principles (idempotency, retries, small payloads) stay the same.

Watch your queues

• Queue length: a growing backlog means workers can't keep up.
• Job duration and failure rate.
• Dead-letter queue size.

Most queue libraries come with a dashboard. Alert when failures spike or the backlog grows steadily. (Monitoring basics.)

Tell users what's happening

If a job's result matters to the user, like an export, show a "processing" state and notify them when it's done, by email, an in-app notification or a real-time update. (Real-time options.)

FAQ

What is a background job?

Work that runs outside the web request, in a separate worker process, so users don't wait for slow tasks like sending emails, processing images or generating reports.

Do I need Redis for background jobs?

Not necessarily. Many libraries can use your existing Postgres database as a queue. Redis-based queues become worthwhile at higher volumes or when you already run Redis.

What does idempotent mean for jobs?

Running the job more than once has the same effect as running it once. It's essential because retries can cause a job to execute multiple times.