Ask
28

Our users are failing on a third-party API and the vendor's status page is entirely green: how do you establish it is them?

About a fifth of our requests to a payment provider have been failing since this morning with timeouts and the occasional 502. The rest go through normally.

Their status page says all systems operational and has done all day. Support have replied once to say they are not seeing any issues.

On our side nothing was deployed, our other integrations are fine, and the failures are spread across our servers rather than coming from one machine.

I am fairly confident it is them, but "I am fairly confident" is not something support will act on and I have already been told they see nothing. What evidence actually moves this, and how do I tell the difference between their problem and something in the path between us?

3 answers Share
Report

Answering anonymously, a moderator will review it first.

  • @partial_outage_pia · 3w ago

    Start from the fact that a green status page and a real outage are entirely compatible, and understanding why tells you what to measure.

    Status pages are typically updated by a human after an incident is declared, against thresholds set by the vendor. Most real-world degradations never cross those thresholds because they are partial: one region, one data centre, one endpoint, one customer tier, or a percentage of requests. A fifth of your requests failing is a textbook partial, and a vendor whose global error rate moved by a fraction of a percent will show green and mean it.

    Which also explains "we are not seeing any issues" - they are probably looking at an aggregate, and you are a rounding error in it.

    So the shape of the evidence you need is specific enough to look up, not statistical. What actually gets acted on:

    • Request ids or trace ids from the failed calls. Most APIs return one in a response header even on errors, and this is the single most valuable thing you can hand over - it lets them find your exact request in their logs rather than looking at a graph.
    • Exact timestamps with the timezone stated.
    • The precise endpoint and method. Partial outages are often one endpoint.
    • Your source region, because regional is the most common shape.

    One ticket with ten request ids beats twenty messages describing a percentage.

    30
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @evidence_first_efe · 3w ago

    To separate them from the path between you, which is your second question and the one that decides who you are even talking to:

    Test from somewhere else. Run the same call from a different network and a different region, a cloud shell, a VPS, your laptop on a phone hotspot. If it fails everywhere, it is them. If it only fails from your infrastructure, the problem is in your path or your egress, and no amount of pushing on their support will move it. This takes five minutes and it is the highest-value thing in this thread.

    Look at where in the request it dies. A timeout during connection establishment points at the network path. A timeout after the request was accepted but before a response points at their processing. Curl's timing breakdown gives you this directly, DNS, connect, TLS, first byte - and the shape of it is genuinely diagnostic.

    Check DNS resolution. Compare what your resolver returns against a public resolver. Getting a different or stale set of addresses is a real and surprisingly common cause of "a fraction of requests fail", because some of your connections are going somewhere they should not.

    Rule out your own retry behaviour. A fifth failing with some 502s can also be you retrying into a pool that is already saturated. Check whether the failure rate correlates with your own request volume, because if it does, part of this is yours.

    26
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @measure_it_yourself · 3w ago

    The durable fix is to stop relying on their page: run your own probe against the endpoints you depend on, from outside your application, on a schedule, and keep the history.

    This costs very little and it changes every future conversation. Instead of "we think you are having problems", you arrive with a graph of your own success rate and latency against their API going back months, with a visible step change at 09:14. That is not arguable, and it is also proof it is not a change on your side, because the probe is not part of your application.

    Two details that make it worth having:

    Probe from more than one place if you can. Two regions turns "it is broken" into "it is broken from here and fine from there", which is exactly the sentence that gets a partial outage taken seriously.

    Record the request id on every probe, successful or not. Then when something goes wrong you already have the identifiers rather than scrambling to collect them while it is happening.

    And the general habit: subscribe to their status page updates, but treat the page as their opinion, not as data. Your own measurement is the only thing that tells you what your users are experiencing, and it is the only thing that ever gets a vendor to look properly.

    1
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report