Ask
27
@pipeline_pars ·

Where do credentials actually belong in a pipeline, given that everything ends up in a log eventually?

My pipeline needs a username and password for a registry, a token for a deployment target, and a key for signing. Right now some of these are variables in the project settings and one is, embarrassingly, in the pipeline file.

I know the file version is wrong. What I am less sure about is whether the settings version is actually safe, given that anybody who can edit the pipeline can also write a step that prints the variable.

What does a defensible setup look like?

4 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @pipeline_pars · 3h ago

    Your instinct is exactly right and it is the thing most teams have not internalised: anybody who can change the pipeline can read every secret the pipeline can reach. Masking hides a value in logs; it does not stop a step base64-encoding it and printing that.

    So the defensible setup is not about hiding values, it is about limiting what each secret can do and who can trigger a run that sees it.

    The layers that actually help:

    • Protected secrets. Most systems let a variable be exposed only to runs on protected branches or tags. That stops somebody opening a merge request from a fork with a step that exfiltrates production credentials — which is the realistic attack, not a leaked log.
    • Scope every credential down. A deploy token that can push one image to one repository is a much smaller problem than an account password. Registry credentials in particular should never be a human's password.
    • Separate environments. Production secrets available only to the production job, not to every job in the file.
    • Rotate on a schedule, and immediately when anybody with access leaves.
    30
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @edge_ekin · 3h ago

    Practical hygiene that costs nothing and catches real mistakes:

    • Never echo a variable to check it exists. Test with [ -n "$VAR" ] instead. The debugging echo is how secrets reach logs.
    • Turn off command tracing around anything that takes a secret as an argument, or pass it on standard input instead of on the command line, since the command line is visible in the process list.
    • Pass secrets as environment variables to the process that needs them, not into a file the workspace keeps. Workspaces get cached and artefacts get uploaded.
    • Scan the repository for committed secrets in a pipeline step. They are already there in more projects than people expect, and history keeps them after the file is fixed.

    And assume anything that ever reached a log is compromised. Rotating is cheap; deciding it was probably fine is how incidents start.

    21
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @acme_asu · 2d ago

    The upgrade that removes most of this problem where it is available: stop storing long-lived credentials at all.

    Most cloud providers and registries now accept a short-lived token that the CI system mints for a specific job, tied to the repository and branch, valid for minutes. Nothing is stored anywhere, there is nothing to rotate, and a leaked log line is worthless by the time anybody reads it.

    It goes by different names — workload identity, OIDC federation, keyless authentication — and the setup is a trust relationship configured once on the provider side.

    Where it is not available, the next best thing is a secrets manager the job authenticates to, so the CI system holds one credential rather than twenty, and the manager logs every retrieval. That audit trail is worth a lot when somebody asks what a departing engineer had access to.

    26
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @queue_qamar · 2d ago

    One organisational point that closes the loophole you spotted: restrict who can change the pipeline definition on protected branches.

    If the file requires review to change, then reading a production secret requires getting a malicious step past a reviewer. That converts a technical control you correctly identified as weak into a process control that is much stronger.

    It is also the control most often missing, because the pipeline file lives in the repository and inherits whatever the repository's rules are.

    14
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report