The request just timed out: moving long work into background jobs and queues

If any work inside an HTTP request takes longer than 30 seconds, that work is in the wrong place. Spreadsheet exports, a 40,000-row import, PDF generation, bulk email, a loop of calls to a third-party API: none of these should run while a browser holds a connection open. They belong in a separate worker process. The shape of the fix is simple enough: accept the request, write a job to a queue, hand back a job id, and make the status queryable. Picking a queue product is the easy part. What actually costs you time is the same job running twice, infinite retries on a permanent error, and a dead-letter queue nobody ever looks at.

Raising the timeout does not help, because the limit is not yours

Knowing where a long request gets cut off is half the work. The Heroku router gives up after 30 seconds and returns an H12, and here is the nasty part: the dyno behind it keeps processing the request, the router has no idea, so it keeps sending new requests to that busy dyno and the problem compounds. An AWS Application Load Balancer defaults to a 60 second idle timeout. Nginx defaults proxy_read_timeout to 60 seconds as well. Behind Cloudflare, your origin has 100 seconds to respond before the visitor gets a 524, and that window is only adjustable on Enterprise plans.

These limits sit in different layers, which is why raising one rarely helps. Pushing nginx to 120 seconds does nothing when the load balancer in front of it still cuts at 60; you have only changed which component reports the failure. It also does nothing for the person watching a spinner for 90 seconds. Long work gets cut somewhere until you take it out of the request path.

Change the request/response contract first

Before adding a queue, the API behaviour has to change. The endpoint that starts a long operation no longer performs it: it creates a record, enqueues a job, and returns 202 Accepted with a job id. The client polls a status endpoint with that id, or subscribes to progress over server-sent events. For work measured in minutes, like a monthly report, the most honest answer is usually the plainest one: "we will email you when it is ready."

There is a real cost on the interface side, because you now have three states to display: queued, running, finished or failed. Teams routinely underestimate this. A queue architecture where the user cannot find out where their report went generates more support tickets than the synchronous timeout it replaced.

You probably do not need new infrastructure for this

The reflex is to install Redis or provision SQS. For most teams Postgres is enough. SELECT ... FOR UPDATE SKIP LOCKED landed in PostgreSQL 9.5 in January 2016, and it lets a worker skip past rows another worker has locked instead of queueing up behind them. Que in Ruby and Oban in Elixir are built on exactly that. Up to a few hundred jobs per second this works fine.

The real advantage of Postgres here is not speed, it is transactional integrity. Enqueue the job inside the same transaction that creates the record and the "job queued but record rolled back" case simply cannot happen. With Redis or SQS you lose that atomicity, and the fix is an outbox table plus a relay process that moves rows into the real queue. There are good reasons to run Sidekiq, BullMQ or SQS: high throughput, isolated worker fleets, a monitoring UI you get for free. "A queue means Redis" is not one of them. Every new infrastructure component comes back to you later as backups, version upgrades and on-call load.

Every job will run twice, so write it that way

Almost every queue offers at-least-once delivery, not exactly-once. The mechanism is a lease: a worker receives a message, and for the length of the lease that message is hidden from other workers. In SQS the visibility timeout defaults to 30 seconds and can be set anywhere from 0 seconds to 12 hours. In Google Pub/Sub the acknowledgement deadline defaults to 10 seconds, can go up to 600, and the client libraries extend it while processing continues.

The consequence is unavoidable. If a job outlives its lease, the message is redelivered and a second copy starts while the first is still running. The same thing happens whenever a worker process dies mid-job. So every job has to be safe to run again. In practice that means deduplicating the outcome against a business identifier: a unique constraint in the database, or an advisory lock taken at the top of the job. The difference between "the email went out twice" and "the customer was invoiced twice" is whether that constraint exists. The payment-side version of this problem is covered in the payment integration article.

Retries raise two questions: which errors, and how far apart

Not every error deserves a retry. Transient ones do: connection timeouts, 502s, 429s, database deadlocks. Permanent ones are already finished on the first attempt, and retrying a validation failure or a declined card twenty times only inflates your logs and buries the real signal. Make that distinction explicit in code, because library defaults tend toward "retry every exception."

For the interval, do not use a fixed delay. Exponential backoff with random jitter is the standard approach, and the full jitter variant compared on the AWS architecture blog produces the least total load, which makes it a sound default for most services. Without jitter, every worker comes back in the same second after an outage and knocks the dependency over again. Sidekiq's defaults are instructive: 25 attempts spread over roughly 21 days, using (retry_count ** 4) + 15 + rand. Reasonable for an archival job. For an order confirmation email, an attempt that lands 21 days later is worse for the customer than the original failure. Set the attempt count per job type rather than inheriting one number for everything.

Put a concurrency limit on jobs that call outward

Adding workers drains a queue faster, right up until the work leaves your network. A pool of 50 concurrent workers hitting a shipping or e-invoicing API that accepts 10 requests per second will produce a wall of 429s within minutes. Layer retries on top and the load goes up, the provider throttles you harder, and the queue grows instead of draining.

The answer is not fewer workers overall, it is a concurrency cap or token bucket scoped to that job type. If the provider sends a Retry-After header, obey it instead of your own backoff formula; their number beats your guess every time. Jobs above the cap wait in the queue, which is fine. When nobody is staring at a screen, a 20 second delay costs nothing.

A dead-letter queue nobody reads is silent data loss

A job that exhausts its attempts has to land somewhere. In SQS, exceeding maxReceiveCount moves the message to a dead-letter queue; in Sidekiq, a job that fails 25 times drops into the dead set and sits there for six months. The point is to stop a poison message from blocking the queue, so one malformed record cannot hold up hundreds of healthy jobs behind it.

Whether this actually helps comes down to one thing: messages arriving in the DLQ must raise an alert, and that alert must have an owner. Otherwise you find out at month end, when somebody notices 300 order notifications never went out. Have the redrive path ready too. After you fix the bug you will not want to recreate 300 jobs by hand.

One queue for everything stops working early

Throwing every job into a single queue causes no trouble in the first weeks. Then one customer starts a 40,000-row import and a password reset email waits 20 minutes behind it. Split queues by latency expectation rather than by domain: work someone is waiting on, work that should finish within minutes, and overnight batch work. Give each group its own worker pool so one cannot starve another.

Do not write batch jobs as single units either. Instead of processing 40,000 rows in one job, have a parent job split the input and enqueue 400 child jobs. That buys you three things: you stop outliving the lease, partial progress becomes visible, and a failure retries one chunk instead of all 40,000 rows.

Pass the id, not the payload

Stuffing the whole object into the job payload is a common mistake. When the job runs five minutes after being enqueued, that snapshot is stale: the order may be cancelled, the price may have changed. Put the identifier in the payload and let the worker read the current record.

Size is another reason. Amazon SQS raised its maximum message size from 256 KiB to 1 MiB in August 2025, but billing is metered in 64 KB units, so a single 1 MB message is charged as 16 requests. The cost of casually fattening payloads grows quietly. There is a security angle as well: job arguments sit in Redis, land in logs, and are readable in dashboards like Sidekiq Web. Secrets and personal data do not belong there, references do. Where secrets should live instead is covered in the secrets management article.

Scheduled jobs are their own trap

Cron expressions work as long as you run exactly one server. The day you scale to two instances, two billing batches start together at 03:00. Run the scheduler in one place, via leader election or a distributed lock, and add an "is this already running" check at the top of every scheduled job, because the next trigger can fire before the previous run finishes.

Time zones are the second trap. A cron configured in local time will, in regions that observe daylight saving, run a job twice one night a year and skip it entirely on another. Schedule in UTC and convert only when displaying times to users.

An unmeasured queue backs up quietly

The metric list is short: queue depth, age of the oldest waiting message, p95 job duration, failure rate, DLQ size, and worker saturation. The one to alert on is not depth, it is the age of the oldest message. Depth swings around and that is normal; "the oldest job has been waiting 15 minutes" is always meaningful.

Watching those numbers without a target attached does not get you far. Commit to something like "95% of export requests finish within 2 minutes" and set the alert against that. For the mechanics of targets and error budgets, see the SLO and observability article. How workers shut down during a release belongs to the same topic: if you do not signal running jobs and wait for them to drain, every deploy leaves half-finished work behind. That side is covered in the zero-downtime deployment article.

Where to start

Pull p95 duration per endpoint out of your logs and list everything above 3 seconds. Moving the top three into the background is usually a few days of work and removes most of your timeout complaints. If you already run a queue, three questions are enough for a review: does running a job twice corrupt data, who gets alerted when something lands in the DLQ, and are attempt counts tuned per job type or still on library defaults. In most projects the answer to the third one is that everything is still on defaults.