Ask
231
@desmond_ruiz ·

docker container or a throwaway vm for agent shell commands, solo dev, 200 runs a day sandboxing

The agent writes and runs shell commands against a checkout of our repo - tests, migrations, the odd script. About 200 runs a day, all triggered by me, no public input anywhere in the path yet.

Right now it runs on my machine with no isolation at all, which I know is silly. The two options I am weighing are a Docker container per run, or a microVM per run using one of the sandbox services.

I am one person. How much isolation is actually warranted here, and what is the cheapest version that is not embarrassing?

9 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @crimp_not_solder · 8mo ago · 3 replies

    The question that decides this is not Docker versus microVM, it is: who writes the text that becomes a command? Right now the answer is you, and the threat model is "the model does something stupid", not "an attacker is trying to escape". A container is entirely adequate for stupid. It stops being adequate the day a customer's ticket text reaches that agent, and then you want a real VM boundary.

    Ephemeral container per run, and specifically:

    • fresh container each run, --rm, no reuse
    • non-root user, --read-only rootfs with a writable tmpfs for the workspace
    • --cap-drop=ALL, --pids-limit, a memory limit
    • never mount the docker socket in, which is the one mistake that turns all of the above into decoration
    • --network=none unless the run needs the network, and if it does, an allowlist proxy rather than open egress

    At 200 runs a day the container start cost is irrelevant and you can build this in an afternoon. Do that, and revisit the day untrusted input enters the picture.

    204
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @crimp_not_solder · 8mo ago

      It is the single most common mistake and it is not obvious, because everything works and nothing warns you. Anything with the socket can start a privileged container mounting the host root, so you have handed it the machine. If a run needs compose, give it docker-in-docker or a nested runner, never the host socket.

      74
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @desmond_ruiz · 8mo ago

      The socket line is the one I needed. My first attempt at this mounted /var/run/docker.sock so the agent could run the compose file, which I now understand was a container with extra steps.

      61
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @onebag_ozzy · 8mo ago · 2 replies

    Whatever you choose, treat credentials as the real boundary rather than the sandbox. The scariest thing in a dev machine agent run is not rm -rf, it is a cloud credential in the environment that lets it delete a bucket, and no container isolation touches that.

    Minimum: a separate scoped token for the agent, read-only where possible, nothing production, and a git checkout it cannot push from. My agent has a deploy key with read access and no push, and the number of times that has saved me from an over-enthusiastic force push is two.

    137
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @desmond_ruiz · 8mo ago

      Read-only deploy key is going in today. My .env is currently mounted into the run wholesale, which on reflection includes a production database URL that nothing in the test suite has any business seeing.

      46
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @printbed_gremlin · 8mo ago

    Practical note from doing this for a year: the container per run is easy, keeping it fast is the annoying part. Rebuilding node_modules in a fresh container every time turned 200 runs a day into a wall of waiting. Cache the dependency layer in the image and mount the package cache as a volume - the workspace stays ephemeral, the caches persist, and runs went from about 90 seconds to 11.

    89
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @listening_lane · 8mo ago

    Add a wall-clock timeout and an output cap while you are building this. Not for security - for the run that starts a dev server in the foreground and sits there until you notice an hour later. timeout 300 and truncating stdout at a few thousand lines has saved me more grief than any of the capability flags.

    47
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @two_vcpu_club · 8mo ago

    MicroVM services are genuinely good and the boot times are in the low hundreds of milliseconds, so it is not the performance argument people expect. The reason I would not start there as a solo dev is that it puts a network hop and somebody else's API between you and your own test suite, and the debugging story when a run behaves differently in there is worse than the problem you were solving. Reach for it when you have untrusted input, which is a real line and you will know when you cross it.

    64
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @film_fridge · 8mo ago

    Worth writing down what the sandbox is for before building it. Mine started as security and turned out to be mostly reproducibility - the value was that every run started from a known state, so a failing test was the agent's fault rather than my machine's. That framing also tells you what to cache and what must be fresh.

    29
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report