315
Reverse proxy returns 502 for one container after every host reboot Networking
Running about a dozen containers behind a reverse proxy. After every host reboot, eleven of them come up fine and one returns 502 until I manually restart that container, after which it works for weeks. The container itself reports healthy in docker ps even while the proxy is returning 502. Proxy config is identical in shape to the others, just a different upstream port. Nothing useful in the proxy logs beyond the 502 itself. What is the usual cause of this pattern?
@chmod_confused · 4w ago · 3 replies
Almost always a DNS caching issue in the proxy combined with startup ordering. The proxy resolves the container name to an IP once at startup, the container gets a different IP on the reboot because it came up in a different order, and the proxy keeps talking to an address that no longer exists. Restarting the container makes it re-register and often gets the old IP back, which is why the manual fix looks like it fixes the container. The real fix is to make the proxy re-resolve: use a variable for the upstream so it resolves per request, or pin the container to a static IP or a user-defined network with a stable alias.
Reply
Report
@chmod_confused · 4w ago
Exactly the mechanism. Add a
depends_onwith a health condition so the proxy starts last, and use the resolver approach so ordering stops mattering at all. Belt and braces because reboots are the one time you are not watching.Reply
Report
@integral_ines · 4w ago
That explains why it is always the same container: it is the one whose start is slowest because it waits on a database. Presumably it comes up after the proxy every single time.
Reply
Report