158

Build args with secrets still show in docker history on old images Infrastructure

We passed an npm auth token as a --build-arg for months. We've since moved to BuildKit secrets, but images built during that window are still in our registry and docker history shows the value in plain text. How bad is this in practice, and is rotating the token enough or do we need to delete every affected tag?

6 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @rollback_rae · 9mo ago · 2 replies

    Rotate the token first, today, before you touch anything else. Once it's rotated the leaked value is worthless and the rest is hygiene rather than an incident.

    Then yes, the value lives in the layer metadata forever for those images. You can't edit history in place, only delete tags and garbage collect. Order I'd work in: rotate, delete affected tags, run registry GC, then check whether the registry replicates anywhere or gets backed up, because that copy needs purging too. Don't spend a day scrubbing images before the rotation is done.

    132
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @static_typing_fan · 9mo ago

      Rotated within the hour. GC found 340 affected manifests. The replication target is the thing I would have forgotten, so thank you.

      34
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @cors_error_cleo · 9mo ago

    For what it's worth, --secret id=npmrc with RUN --mount=type=secret doesn't touch the layer at all, so you're on the right pattern now. The one gotcha is that if you cat the mounted secret into a file inside the image during the build, you're back where you started. Mount it, use it, never copy it.

    88
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @hot_heap_hana · 9mo ago · 2 replies

    Add a check so it can't recur. docker history --no-trunc piped through a few regexes for ghp_, npm_, AKIA and a generic high-entropy match runs in seconds and fails the build. Ours caught a teammate pasting a token into a Dockerfile comment two weeks after we added it.

    45
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @why_wren · 9mo ago

      Or use one of the off-the-shelf image scanners in fail-the-build mode. Same idea, less regex to maintain.

      11
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @shortcut_sam · 9mo ago

    Multi-stage helps the general case too. The builder stage never gets pushed, so anything you leak there stays on the machine. Doesn't fix your existing images but it narrows the blast radius going forward.

    17
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report