Ask
27

A self-hosted app's interface keeps saying the connection was lost, but the app itself is running fine

I put a self-hosted application behind my reverse proxy. It loads, I can sign in, and the pages work - but the interface repeatedly shows a banner saying the connection has been lost and it is trying to reconnect.

The server is up the whole time. Nothing in its log looks like a crash. If I reload, it works again for a while and then the banner comes back.

What is actually disconnecting, given that the application is clearly still there?

9 answers Share
Report

Answering anonymously, a moderator will review it first.

  • @gpu_gorkem · 4w ago

    One thing worth ruling out before touching the proxy, because it costs a minute and it is occasionally the whole answer: whether the problem started with a version upgrade.

    A live-channel regression in a release is not rare, and the symptom is exactly this: everything works, the banner appears, nothing in the log. If the timing lines up with an update, checking the project's issue tracker for that version, and briefly rolling back to confirm, tells you whether you are configuring around a bug.

    That is worth doing first because every proxy setting you change while chasing an upstream bug is a setting you will not remember to change back.

    22
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @selfhost_sedat · 4w ago · 2 replies

    So the fixes, in the order to try them:

    1. Pass the upgrade headers for that route, and set the protocol version the proxy uses to the backend to one that supports it.
    2. Raise the read timeout substantially for that route - minutes, not seconds - or configure keepalive pings if the application offers them.
    3. Turn off response buffering on that route if the application uses an event stream.

    And there is a fourth option that is often the easiest and gets overlooked: many applications can switch transports. If the app supports server-sent events as an alternative to WebSockets, setting that is a single environment variable and event streams pass through far more proxies unmodified, because they are ordinary HTTP responses that simply never end.

    If you are behind a content delivery network as well as your own proxy, check its settings too: several buffer or terminate long connections regardless of what your proxy says.

    26
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @read_the_proxy_log · 3w ago

      Timeouts are the other half. A default read timeout of sixty seconds will kill an idle live channel every minute, exactly.

      11
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
  • @selfhost_sedat · 4w ago · 3 replies

    What is disconnecting is the live channel, not the application. Modern interfaces keep a long-lived connection open to receive updates: a WebSocket, or a server-sent event stream - and it is that connection dropping, not the HTTP requests.

    That is why everything else works. Loading a page is a short request that succeeds; the banner is about the persistent one.

    And the usual culprit is the proxy, because a long-lived connection needs treatment that ordinary requests do not:

    • The upgrade has to be forwarded. A WebSocket starts as an HTTP request carrying Upgrade and Connection headers, and a proxy that does not pass them through downgrades the handshake and the connection never establishes.
    • The read timeout kills it. Proxies close connections that have been idle for some period, often sixty seconds by default. A live channel that is quiet for a minute is exactly that, so it gets cut, reconnects, and gets cut again. The interval between banners is usually a giveaway.
    • Buffering breaks event streams. A proxy that buffers responses holds the stream instead of passing events through.
    30
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @read_the_proxy_log · 3w ago · 2 replies

      Before changing proxy config, look at the access log for that route. A failed upgrade shows up as a 400 or a 200 that closes immediately, repeating on a fixed interval, which is the reconnect attempt.

      That confirms the diagnosis in ten seconds and means you are not editing config against a guess. The interval in the log is also the client's retry timer, which is a useful thing to recognise elsewhere.

      23
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
      • @selfhost_sedat · 2w ago

        Good addition. The repeating pattern at a fixed interval is the giveaway that it is a client retrying rather than a server crashing.

        13
        Share
        Reply

        Answering anonymously, a moderator will review it first.

        Report
  • @cloudflared_now · 3w ago

    Worth knowing that a tunnel handles this without configuration, since it carries the connection rather than proxying it hop by hop. Not an argument to move if your proxy works, but if you are fighting upgrade headers on three separate apps it is a legitimate way out.

    16
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @esp_ece · 4w ago · 2 replies

    A quick way to see which layer is failing: open the browser's network tab, filter to the WebSocket or event-stream connection, and watch it.

    You will see either the handshake failing outright: which points at the upgrade headers - or the connection establishing and then closing after a consistent interval, which points at a timeout, and the interval tells you whose timeout it is.

    That single observation separates the two main causes without changing anything.

    1
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @cloudflared_now · 3w ago

      The network tab check is the fastest first move and it needs no server access at all.

      8
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report