Qamar

One practical habit: create the host directory yourself with the right ownership before the first run, rather than letting the engine create it.

When the directory does not exist, the engine creates it owned by root, the container starts as a non-root user, and you get this failure on a brand new install with nothing obviously wrong. Creating it in advance with the intended owner makes the setup reproducible and documents the requirement.

One organisational point that closes the loophole you spotted: restrict who can change the pipeline definition on protected branches.

If the file requires review to change, then reading a production secret requires getting a malicious step past a reviewer. That converts a technical control you correctly identified as weak into a process control that is much stronger.

It is also the control most often missing, because the pipeline file lives in the repository and inherits whatever the repository's rules are.

The complexity worry is fair, and there is a middle option people miss: use your database as the queue.

A table of jobs, a worker process that claims rows with a conditional update, and a retry column. No broker, no new infrastructure, transactional with the rest of your data — which is a genuine advantage, because enqueueing a job in the same transaction as the change that caused it removes an entire class of bug where one happens and the other does not.

Several frameworks now ship exactly this as a built-in, precisely because most applications never needed the broker.

When you outgrow it: very high throughput, or fan-out to many consumers. That threshold is far higher than people assume. A database-backed queue handles a workload most applications will never reach, and the operational simplicity is worth a lot.

Decide on what happens when it fails, because that is the axis the options actually differ on.

A thread or a fire-and-forget task in the web process. The work dies with the process. A deploy, a crash or a restart loses it silently, and nothing tells you. Acceptable only when losing the task is genuinely fine — warming a cache, sending a nice-to-have notification.

A durable queue with separate workers. The task is written down before the request returns. It survives restarts, it retries on failure, it can be inspected, and failures land somewhere visible. This is what you want the moment a lost task means a customer does not get something they paid for.

A scheduled script. Simple and correct for genuinely periodic work with no per-request trigger. It is not a queue and does not become one; the moment you find yourself writing a table of pending work for the script to scan, you have written a worse queue.

So: is losing this task acceptable? That question sorts almost every case.