Use a background job when work must continue independently of the browser request, take longer than the request should wait, or retry after a temporary failure. A queue makes that work durable and controllable. Reliability still depends on how the job handles duplicate execution, partial success, and a provider that stops accepting requests.
Give each job a business purpose
Good initial candidates include generating an export, sending a transactional email, importing a file, or synchronizing a record with a third-party system. Avoid moving an operation into a queue merely because it contains an external call.
If a user needs an immediate answer to know whether an action succeeded, define the synchronous boundary carefully. An export request can return once its job is durably accepted. A booking confirmation cannot claim a space is reserved merely because a reservation attempt is waiting in a queue.
Store a job's identifier, tenant, operation type, relevant resource identifiers, and state. Prefer references over unnecessary copies of sensitive data. If a job needs a historical snapshot, state that explicitly rather than accidentally processing whichever version exists later.
The user interface should distinguish accepted, running, completed, and failed work. A success toast that means only “we added it to a queue” creates confusion when processing later fails.
Design for execution more than once
A worker can complete an external action and crash before recording completion. The queue may then deliver the job again. Treat duplicate execution as an expected scenario rather than an exceptional incident.
Idempotency means repeating the same logical operation does not create another business effect. For an import row, use a stable business identifier and a database uniqueness constraint. For a payment API, use the provider's supported idempotency mechanism and understand its scope and retention behavior.
Stripe's idempotent request documentation explains its retry semantics, including differences between API generations. Do not assume every external provider implements the same contract.
A local “completed” flag alone cannot protect an external action across a crash. There is still a window between the provider accepting the operation and the worker updating that flag. Combine provider support, local state, and reconciliation where the business effect requires it.
Close the database-to-queue gap
Consider a committed invoice followed by a failed attempt to enqueue its notification. The invoice exists, but no worker knows to send it. Reversing the order can create a queued notification for an invoice whose transaction later fails.
A transactional outbox addresses this gap. Save the business change and an outbox record in the same database transaction. A separate publisher reads committed outbox entries and submits work to the queue.
The publisher can crash after enqueueing and before marking an entry published. Therefore the receiving job must still tolerate duplicates. The outbox makes lost publication recoverable; it does not make the entire distributed operation execute exactly once.
PostgreSQL's transaction tutorial documents the all-or-nothing database boundary. Keep the transaction short and avoid holding database locks while waiting for a remote API.
Retry errors that can recover
A network timeout, temporary dependency failure, or provider throttling response may justify retrying. An invalid recipient address or malformed import row usually requires a correction instead.
Use bounded attempts with increasing delays and jitter where appropriate. Backoff gives a recovering dependency breathing room; jitter prevents many jobs retrying at the same instant. BullMQ documents automatic retry and backoff options, but configure them around the operation's failure modes.
A timeout leaves an ambiguous result: the request might have reached the provider. Check the provider state or repeat with the same supported idempotency key before creating a new operation.
After the retry budget is exhausted, retain enough information for diagnosis and controlled replay. A failed-job list or dead-letter workflow needs an owner, an alert policy, and a runbook. Infinite retries can conceal permanently broken data while consuming capacity.
Separate concurrency from rate limiting
Concurrency limits the number of jobs executing simultaneously. Rate limiting limits how many operations begin or occur over an interval. A worker can obey a concurrency cap and still exceed a provider's request budget when jobs finish quickly.
Use both when needed. Bound concurrent database work to protect connections and locks. Bound outbound request rate to respect the external service. Honor the provider's documented throttling signals, and account for the fact that one job may make several API calls.
BullMQ's rate limiting guide describes a limiter shared by workers on the same queue. Do not assume adding worker replicas creates independent rate allowances. The guide also warns that older group-key examples do not apply to newer major versions.
For a multi-tenant product, global limits and tenant fairness solve different problems. One large import should not starve every customer's notification. Separate workloads or apply a supported tenant-aware scheduling approach when fairness matters. See multi-tenant architecture.
Handle stalled workers and deployment shutdown
A job can stop progressing because the process crashed or failed to maintain its processing lock. Queue systems may reclaim that job. This is another reason every consumer needs duplicate protection.
In Node.js, sustained CPU work can interfere with the event loop. BullMQ's stalled-job guidance discusses this failure mode and separate processors. Increasing a timeout without understanding the blocking work can merely delay recovery.
During deployment, stop accepting new work, let active jobs finish within a defined shutdown window, and allow safe reclamation for unfinished jobs. Version job payloads when producers and workers can run different releases at the same time.
For long imports, checkpoint progress at meaningful boundaries. A retry should resume or safely repeat a batch rather than forcing an entire file to restart without a clear record of completed effects.
Monitor outcomes and recovery
Queue depth alone is not enough. A short queue can still contain one very old, important job. Track oldest waiting age, processing duration, retry rate, terminal failures, and the time from user request to business completion.
Attach a correlation identifier from the request to the outbox, queue job, and provider operation. Include tenant context with controlled access to logs. Exclude credentials and unnecessary payload contents.
Rehearse failure deliberately: crash a worker after the external call, stop the provider, enqueue a duplicate, and deploy while work is active. Verify that users see an accurate state and operators can recover without manually guessing which effects already happened.
The same principles apply to webhook processing. If your MVP architecture names a queue but cannot explain replay and duplicate handling, the design still needs work.