Ask
28

One container runs about ten times slower than it should on this machine - is that emulation, and what is the actual fix?

Most of our containers are fine. One of them takes roughly forty minutes to do a job that takes about four on a colleague's older laptop.

It runs. Nothing errors. It is simply extremely slow, and only this one image.

I get a warning at start-up about the requested platform not matching the host, which I have been ignoring because everything still worked.

I assume this is the architecture difference and something is being emulated, but I do not understand what is being emulated or why one image is affected and others are not. And I do not know whether the fix is on my side or in how the image was built.

13 answers Share
Report

Answering anonymously, a moderator will review it first.

  • @multiarch_build · 3w ago · 2 replies

    If it is your own image, build it for both architectures and stop thinking about it. The tooling for this is built in and it took me an afternoon once.

    What that involves in practice:

    Build a multi-architecture image so both variants live under one tag and every machine pulls the right one automatically. Your colleagues on older hardware and your CI keep working exactly as before.

    Build the native variant natively where you can. Cross-building under emulation in CI is the slowest possible arrangement and it is where people give up on this. If your CI offers runners on the newer architecture, use them for that half.

    Watch the base image. A multi-architecture base does not guarantee everything on top is, one dependency that only ships one architecture will force the whole thing.

    If it is not your image, the practical order is:

    1. Look for a tag that publishes your architecture
    2. Look for a community image that does
    3. Build it yourself from their source
    4. Accept the emulation and move that one job somewhere else - CI, a remote machine, a colleague's laptop

    Option four is a real answer for a job you run occasionally. It is a bad answer for something in your daily loop, and forty minutes sounds daily.

    26
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @ci_matches_prod · 2w ago

      Building for both architectures is right and there is a cost nobody mentions: your build time roughly doubles, and if you are building on the wrong architecture yourself, the cross-built half is slow to produce as well.

      Worth doing anyway. Worth knowing before you promise it will take an afternoon.

      17
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
  • @platform_flag · 2w ago

    The thing that keeps this hidden: emulation does not announce itself. The container runs, the tests pass, everything is correct, it is just slow, and slow gets attributed to the machine or the job or the day.

    Check the image architecture rather than reasoning about it. If one image in your set was built for a different architecture from the rest, that is your forty minutes, and it is usually a base image somebody pinned by digest two years ago.

    24
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @emulation_tax · 3w ago · 2 replies

    That warning is the whole answer and it should not be ignored - it is telling you the image contains binaries for a different processor architecture, so every instruction is being translated as it runs.

    Why one image and not the others. Your other images almost certainly have builds for your architecture published, so you silently pull the right one. This one only publishes the older architecture, so you get that build and it is emulated.

    Why the slowdown is so large. Translation cost is not uniform. Ordinary application logic takes a noticeable hit. Anything doing heavy numeric work, tight loops, or crypto takes a much worse one, because those are exactly the instructions that translate badly. A ten-fold difference is at the severe end but it is entirely plausible for a compute-heavy job, which is also why people report wildly different numbers for the same setup.

    How to confirm in ten seconds. Inspect the image and look at its architecture, or run something inside the container that prints the machine type. If it reports the older architecture while your host is the newer one, you have it.

    Where the fix is. Usually in the image, not in your machine. Check whether the upstream publishes a build for your architecture under a different tag - many projects added them late and left the default tag alone. If it does, pinning the right tag is the entire fix and it costs nothing.

    30
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @ci_matches_prod · 2w ago

      Which architecture does your CI build on? That is where most of these get baked in.

      9
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
  • @platform_flag · 3w ago

    Multi-architecture images fix it properly, if you control the build. If you do not, pinning a matching tag is the short version.

    10
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @rosetta_confused · 3w ago · 2 replies

    The bit that confused me for months: your colleague's older laptop being faster is not surprising once you know what is happening, and it is the strongest hint you have. A native run on modest hardware beats an emulated run on fast hardware, every time.

    So "my newer machine is slower than their old one" is almost never a hardware problem.

    18
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @platform_flag · 3w ago

      That comparison is the diagnostic, really. It rules out most of what people check first.

      12
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
  • @rosetta_confused · 2w ago

    Ten times slower is about right for the emulated case, in my experience. Not two times.

    7
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @one_dep_ruined_it · 2w ago

    Ten times is generous, mine was closer to twenty on anything doing real computation.

    7
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @native_deps_nil · 3w ago · 3 replies

    One thing worth checking before you rebuild anything: it may be one dependency rather than the whole image.

    A common shape is an image that is nominally built for your architecture but installs a compiled package that is not available for it, so the package manager falls back to a source build or, worse, the runtime falls back to a pure interpreted implementation of something that is normally compiled. The container is native, one library inside it is not, and that library is doing all the work.

    How to spot it: look at the build log for anything compiling from source that you expected to arrive prebuilt, and look for warnings about a wheel or binary not being found for your platform. Numeric libraries, image processing, cryptography and anything with bindings to a C library are the usual suspects.

    The fix there is different, pin a version that publishes for your architecture, or install the system library so the binding finds it.

    General habit worth adopting after this: stop ignoring the platform warning. It is one line and it appears exactly when something has silently changed cost by an order of magnitude. Treating it as noise is how a four minute job becomes forty and nobody notices for a month.

    1
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @one_dep_ruined_it · 2w ago · 2 replies

      Can confirm the one-dependency shape and it is worth checking first because it is five minutes rather than a rebuild. Ours was a single package pulling a prebuilt binary for the wrong architecture, inside an image that was otherwise entirely native.

      Nothing warns you, because the image itself is fine. Only that one process pays the tax, and if it happens to be the process doing the work, the whole job is ten times slower.

      20
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
      • @native_deps_nil · 2w ago

        Exactly it. And the giveaway is that the container starts instantly and then crawls, rather than being slow throughout.

        12
        Share
        Reply

        Answering anonymously, a moderator will review it first.

        Report