Ask
148
@arraysformula ·

analytics says 41% activation, my own sql says 26% — which one is lying

Same definition on both sides, or so I thought: signed up and completed the first import within seven days. The product analytics tool says 41%, a query against my own database says 26%.

That is not a rounding difference, it is the difference between a product that works and one that does not. I have spent two evenings on it and found nothing conclusive, and I am now suspicious of every number in the dashboard.

Where does this gap usually come from?

8 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @starschema_sy · 4h ago

    Third possibility that nobody has said yet: the seven day window. Your SQL probably applies it strictly from signup timestamp. The tool may be applying it to a cohort period, or in a different timezone, or counting anyone who ever imported and happened to sign up in the window. Timezone alone will move a seven day funnel by a few points.

    Read the tool's definition of the funnel step order too. Some of them count a conversion if both events happened in the window regardless of order, which quietly includes people who imported before they signed up, which sounds impossible until you remember your own test accounts exist.

    97
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @visa_run_val · 3h ago · 2 replies

    The other classic is identity stitching. Anonymous browsing gets one id, signing up creates another, and the tool merges them on identify. If someone signs up on their phone and imports on their laptop, or clears cookies between the two, the tool may count two people where your database counts one. Depending on which side of the join the merge fails, you get either an inflated or deflated funnel.

    Check one specific user by hand. Pick five accounts you know completed an import, and find each of them in the analytics tool. Five manual traces will usually show you the mechanism in twenty minutes, which is faster than reasoning about it.

    118
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @interval_ida · 2d ago

      Manual tracing is the right move and I would add: pick your five from the disagreement, not at random. Specifically accounts your SQL says did not activate. If the tool has them as activated, you learn more in one lookup than in a week of theorising.

      54
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @stdlib_stef · 3h ago · 2 replies

    Almost certainly your denominators differ, and the usual culprit is that the analytics tool never saw some of your signups at all.

    Browser extensions block requests to known analytics domains, so a percentage of your users simply do not register on the front end. It is not small — routing events through your own domain instead of the vendor's typically recovers something in the region of a tenth to a third of events depending on your audience, and a developer-heavy audience is at the top of that range. If your signups are blocked but your imports are tracked server side, or the other way around, you get exactly this shape of discrepancy.

    Fix: send the events that matter from your server, and put a reverse proxy on your own domain in front of the client-side ones. Do not name that subdomain anything containing analytics, tracking, telemetry or the vendor's name, because those get blocked too.

    134
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @skillet_sara · 2d ago

      The direction matters for diagnosis. Blocked front-end signups shrink the denominator, which inflates activation — which is exactly what OP sees, with the tool reporting the higher number. That is a good sign the theory is right.

      62
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @bedroomsynth · 2d ago

    Trust the database. It is the only source that knows what actually happened. Use the analytics tool for shape and direction, never for a number you would put in an email to anyone.

    88
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @tripodwobble · 2d ago

    Had the identical thing and mine turned out to be even dumber than any of these: the client event fired on the import button click, and the server only recorded the import if the file actually parsed. About a fifth of imports failed on a malformed file. The tool was counting intent and my database was counting success, and honestly the gap between those two numbers was the most useful thing I learned that month.

    71
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @proof_pilar · 3h ago

    Practical policy that ended this argument for me: one metric, one source, written down. Anything that touches money or a decision comes from a SQL query saved in the repo with a comment explaining the definition. Everything else can come from the tool.

    It sounds bureaucratic for a one person company and it takes about an hour to set up, but it means you stop rediscovering the same discrepancy every quarter. Also exclude your own accounts and your test emails in the query itself rather than remembering to filter, because you will forget.

    79
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report