Ask
28

Our users are failing on a third-party API and the vendor's status page is entirely green — how do you establish it is them?

About a fifth of our requests to a payment provider have been failing since this morning with timeouts and the occasional 502. The rest go through normally.

Their status page says all systems operational and has done all day. Support have replied once to say they are not seeing any issues.

On our side nothing was deployed, our other integrations are fine, and the failures are spread across our servers rather than coming from one machine.

I am fairly confident it is them, but "I am fairly confident" is not something support will act on and I have already been told they see nothing. What evidence actually moves this, and how do I tell the difference between their problem and something in the path between us?

2 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @partial_outage_pia · 5h ago

    Start from the fact that a green status page and a real outage are entirely compatible, and understanding why tells you what to measure.

    Status pages are typically updated by a human after an incident is declared, against thresholds set by the vendor. Most real-world degradations never cross those thresholds because they are partial: one region, one data centre, one endpoint, one customer tier, or a percentage of requests. A fifth of your requests failing is a textbook partial, and a vendor whose global error rate moved by a fraction of a percent will show green and mean it.

    Which also explains "we are not seeing any issues" — they are probably looking at an aggregate, and you are a rounding error in it.

    So the shape of the evidence you need is specific enough to look up, not statistical. What actually gets acted on:

    • Request ids or trace ids from the failed calls. Most APIs return one in a response header even on errors, and this is the single most valuable thing you can hand over — it lets them find your exact request in their logs rather than looking at a graph.
    • Exact timestamps with the timezone stated.
    • The precise endpoint and method. Partial outages are often one endpoint.
    • Your source region, because regional is the most common shape.

    One ticket with ten request ids beats twenty messages describing a percentage.

    30
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @evidence_first_efe · 5h ago

    To separate them from the path between you, which is your second question and the one that decides who you are even talking to:

    Test from somewhere else. Run the same call from a different network and a different region — a cloud shell, a VPS, your laptop on a phone hotspot. If it fails everywhere, it is them. If it only fails from your infrastructure, the problem is in your path or your egress, and no amount of pushing on their support will move it. This takes five minutes and it is the highest-value thing in this thread.

    Look at where in the request it dies. A timeout during connection establishment points at the network path. A timeout after the request was accepted but before a response points at their processing. Curl's timing breakdown gives you this directly — DNS, connect, TLS, first byte — and the shape of it is genuinely diagnostic.

    Check DNS resolution. Compare what your resolver returns against a public resolver. Getting a different or stale set of addresses is a real and surprisingly common cause of "a fraction of requests fail", because some of your connections are going somewhere they should not.

    Rule out your own retry behaviour. A fifth failing with some 502s can also be you retrying into a pool that is already saturated. Check whether the failure rate correlates with your own request volume, because if it does, part of this is yours.

    26
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report