Ask
74
@rack_unit_12 ·

Ping to our HQ firewall times out from one home connection but tracert to the same IP completes — nothing arrives in the firewall capture Whose Fault?

Managed firewall at a customer HQ. Staff work from home over SSL VPN plus a couple of browser-delivered apps. Since this morning a handful of users can't reach HQ at all from their home connections. Not slow — nothing at all.

From an affected laptop, at home:

  • general internet is completely fine, every other site and service works
  • VPN client, HTTPS to the HQ public IP, and plain ping <HQ public IP> all time out
  • tested by IP rather than hostname, so this isn't DNS
  • packet capture on the HQ firewall shows nothing arriving from that user's public IP
  • tracert <HQ public IP> from that same laptop appears to complete, and I can see those probes in the firewall capture
  • same laptop tethered to the user's phone: ping, VPN and HTTPS all work instantly

Already done: rebooted the ISP router, rebooted the laptop, ipconfig /flushdns, arp -d, and reproduced on a second device on the same home network.

I'm fairly confident this is a path problem out in the ISP, but "I think it's your ISP" is not something a customer accepts from the person who manages the firewall. Two questions. What test actually nails it down? And what would explain ping failing while a traceroute to the same address gets through?

13 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @immutable_infra · 14h ago · 3 replies

    Start with the contradiction itself, because in my experience it usually isn't real.

    On Windows, tracert sends ICMP echo requests with the TTL going 1, 2, 3 and so on, and builds the path out of the time-exceeded replies from routers along the way. Microsoft's own reference page for the command spells that out. The crucial part: the final line only becomes a genuine hit when the destination itself answers with an echo reply. Which means if a tracert truly completed to that address, a plain ping to the same address should also work, because it is the same message type going to the same place.

    So verify two things before theorising:

    1. Run tracert -d <ip> and read the last line properly. Does it print your destination address followed by "Trace complete", or does it run out at 30 hops with rows of asterisks after the last ISP router? A trace that dies immediately past the provider edge looks complete at a glance, especially in a screenshot.
    2. In your HQ capture, check what you actually matched. Are those echo requests addressed to your firewall with the user's public IP as source, or something else that happens to be ICMP and happens to be near the right timestamp?

    Most of the time one of those two dissolves the mystery entirely.

    47
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @blown_cap · 2h ago

      The other version of this mistake: tracert resolved a name to a different address than the one you pinged, so you compared two different destinations. -d and an explicit IP on both commands, always.

      8
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @sql_window_fn · 2h ago

      Count the hops as well. If the trace stops at hop 12 and hop 12 resolves into the ISP's own range, it never reached anything of yours — you're looking at the edge of their network, not at your firewall.

      14
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @query_quinn · 3h ago · 2 replies

    Whatever the cause turns out to be, change your instrument. ICMP tells you less than any other test you could run and a large slice of the internet treats answering it as optional.

    From the client:

    Test-NetConnection <hq-ip> -Port 443
    Test-NetConnection <hq-ip> -Port <vpn tcp port>
    

    That attempts an actual TCP handshake, which is the traffic you care about, and it prints the source address it used, which you'll want for the ticket. If the VPN is UDP you can't get a useful answer from one side alone, so pair it with the HQ capture and note the exact time you ran it.

    If TCP 443 also produces nothing at HQ while tracert probes do show up, that is a far more interesting sentence to put in front of an ISP than "ping doesn't work" — because you've eliminated the easy dismissal that somebody is just filtering ICMP.

    31
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @labcoat_lior · yesterday

      And run the identical pair of commands on the phone hotspot within the same minute. Two outputs, same laptop, same target, one works and one doesn't, timestamps a minute apart — that pair is the single strongest artifact you can hand anyone, and it's five minutes of work.

      9
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @finops_reyna · 15h ago

    There is one genuine difference between the two if you want to chase it in a capture.

    The echo request that finally lands on the destination during a traceroute arrives with a tiny TTL, because it was set to exactly the hop count on the way out. A normal ping arrives with whatever is left of 128. If some device in the path, or in front of your firewall, makes a decision based on that field, it would treat the two differently.

    I wouldn't lead with this — it's uncommon — but checking costs one line. In your HQ capture, look at the IP TTL on the tracert probes you can see. If they land with a TTL of 1 and nothing at all from that source arrives otherwise, you've got something concrete and unusual to point at rather than a hunch.

    19
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @epsilon_eddie · 23h ago · 3 replies

    Ping failing while traceroute succeeds is nearly always a firewall policy dropping echo requests while permitting the traceroute probes. I'd go through the policy on your side first.

    22
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @errwrap_elle · 6h ago

      That reasoning holds on Linux, where traceroute defaults to UDP probes at high ports — different protocol, different rule, different outcome, and the classic explanation for exactly this symptom.

      Windows tracert is ICMP echo, same as ping. A policy that drops echo requests drops both. And it can't be an HQ-side policy in any case: the capture shows nothing arriving. You cannot drop what never reached you. Whatever is happening is upstream of the interface being captured on.

      26
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @epsilon_eddie · 12h ago

      You're right, I had the Linux behaviour in my head. If it's Windows tracert then it's echo-based and my theory doesn't survive contact with the capture.

      7
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @orm_tamsin · 2h ago

    Worth putting a question to the HQ side too, since "never arrives" cuts both ways.

    If there is DDoS scrubbing, a transit provider filter, or an upstream ACL sitting in front of that firewall, traffic can be dropped before it ever reaches the interface you're capturing on — and your capture would look exactly like it does now. Ask whoever provides HQ transit whether that source prefix, or the whole consumer ISP range it lives in, is currently being filtered, scrubbed or rate limited.

    It's a five minute question, and it's the only theory on this page that explains "nothing in the capture" without involving the client's ISP at all. Rule it out before you send the customer down the road of arguing with their provider.

    13
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @wanderport · yesterday · 2 replies

    Cheap local checks that occasionally explain "it never left the house":

    • route print -4 on the laptop. If the home LAN range overlaps the HQ range, or a previously uninstalled VPN client left a persistent route behind, traffic to that destination goes out the wrong interface and dies locally. You would never see it anywhere upstream and no ISP would ever find it.
    • Check the WAN address on the ISP router. If it's inside 100.64.0.0/10, the customer is behind carrier-grade NAT. That's a shared path, and shared paths are exactly the kind of thing that break for one destination prefix while everything else keeps working.
    • If the router exposes a connection or session table, look at whether entries toward the HQ address are being created at all. Created-then-nothing and never-created point at different people.
    16
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @crypt_cass · 2h ago

      Leftover persistent routes from a removed VPN client are more common than they have any right to be. Read the persistent routes section at the bottom of route print -4 specifically — it's easy to skim past it.

      7
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @otis_haverford · yesterday

    On the customer conversation, which is half your actual problem: don't say "I think it's the ISP".

    Hand them one page. Same laptop, same target, same minute, two networks, one works. Then a specific ask, phrased for the ISP rather than for the customer: "from the access node serving this line, is <HQ IP> reachable, and is anything filtering or rate limiting toward it". A named place, a named target, a yes/no answer.

    Customers accept a documented handover far more readily than a verbal opinion from the person they're paying to manage the firewall. It also protects you if it turns out to be something else entirely, because the page records what you tested rather than what you concluded.

    10
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report