2,300 bullmq jobs stuck in active after a deploy, lockduration is default Worker Setup

Both, but mostly the deploy. Two separate problems that look like one.

The lock. Your job runs three times longer than lockDuration. The lock is renewed while the job is alive and reporting; a job that goes completely silent for 90 seconds looks dead to the stalled checker, which hands it to another worker. Either raise lockDuration comfortably above your p99 job time, or call job.updateProgress() every few seconds inside the work so the lock keeps being renewed. The second is better because it also gives you progress for free.

The deploy. Your old pods were killed with jobs in hand. Handle SIGTERM: stop accepting new jobs, await worker.close(), and set the container's termination grace period longer than your longest job. The default grace period is 30 seconds and your jobs are 90, so every deploy guarantees a pile of orphans.

Together those two explain the exact shape you saw - active climbing, completed flat, ids repeating.

For 90 second work, chunk it. Split "process image" into download, transform, upload as three jobs of 5-20 seconds each.

Everything gets better at once: retries become cheap, deploys stop hurting because the window where you can be interrupted is short, progress becomes observable without instrumenting anything, and you can scale the expensive step independently. Long jobs are a design decision, not a requirement.

Separately, make the work idempotent so a stalled re-run is boring rather than dangerous. If the transform writes to a deterministic key in object storage, running it three times costs a bit of CPU and nothing else - no duplicate rows, no duplicate emails, no support ticket.

Worth knowing before you tune anything: maxStalledCount defaults to 1, so a job that stalls twice is moved to failed rather than looping forever. That is why some of yours ended up failed instead of running six times - the framework already saved you from an infinite loop. Raise it only when you understand why they stalled.

Flush Redis and requeue from your own database. Fastest way out of a stuck queue and you know exactly what state you are in afterwards.