Ask
341
@thesis_wrangler ·

a service_role client in a shared db module returned another org's invoices Multi-Tenant

Writing this up because it was entirely preventable and I want to know how other people structure against it.

One dashboard query was slow. Someone swapped the client in lib/db.ts for one built with the service role key because "RLS was making the plan bad". That module is imported by 30 files. Nine days later a customer opened a ticket with a screenshot of an invoice list containing 84,120 rows across 611 organisations, including names of companies that are their competitors.

Nobody was malicious, the diff was four lines and reviewed. What is the structural fix - not "be more careful"?

9 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @wren_oyelaran · 2mo ago · 3 replies

    Make the dangerous client hard to reach and easy to spot:

    • One file, one export, named so it cannot be mistaken: adminDb from lib/admin-db.ts. Never a default export, never re-exported.
    • An eslint no-restricted-imports rule that fails the build for any file outside lib/jobs/** importing it. This is fifteen lines of config and it is the actual fix.
    • Keep the tenant predicate in the query even when using it. Defence in depth costs you one where clause.

    And stop treating "RLS makes the plan bad" as a reason to bypass RLS. It is almost always a bare function call in the predicate, which is a two-character fix, not an architecture change.

    49
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @thesis_wrangler · 2mo ago

      The lint rule went in the same day. What stings is that the original slow query was the auth.uid()-not-in-a-subselect thing, so the entire incident traces back to a missing pair of parentheses.

      14
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @arrayformula_al · 2mo ago

      Add the test too: seed two orgs, run your list endpoint as a user in org A, assert zero rows belong to org B. Run it against the real code path, not the query builder. That test would have failed on the PR that introduced this, which is worth more than any review checklist.

      11
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @supa_okonkwo · 2mo ago

    One Postgres detail people miss: enabling RLS does not apply to the table owner. If your migrations run as the owner and your app connects as the owner, you have policies that do literally nothing.

    alter table invoices force row level security;
    

    That closes the owner hole. It does not stop a role with BYPASSRLS, which is what a service role is, so it is not a fix for your incident - but it is worth checking that your policies are on at all before you trust them.

    26
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @costbasis_carl · 2mo ago

    Since you asked about structure and not comfort, one more: you now have a disclosure to make, and the customer who found it will ask whether anyone else saw their data. Make sure you can answer that from logs. Being able to say "three requests, all from your own account, here are the timestamps" is the difference between a rough week and losing the account.

    22
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @drainfield_dez · 2mo ago · 2 replies

    Different angle: your request path should not have credentials that can bypass RLS at all. Give the web process a role without BYPASSRLS and put the admin credential somewhere the web process cannot read it - a separate worker with its own environment.

    Then "someone imports the wrong client" fails at connection time in development instead of succeeding quietly in production. Permissions you cannot accidentally hold are better than permissions you promise not to use.

    34
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @wren_oyelaran · 2mo ago

      This is the better version of what I wrote. A lint rule is a speed bump; a credential the process cannot read is a wall. The only cost is that your local dev setup now needs two connection strings, which is a fine price.

      13
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @layoverlurker · 2mo ago

    We keep the admin client in a package that is not in the web app's dependency list at all. You cannot import what is not installed. Slightly annoying in a monorepo, extremely effective.

    18
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @keg_line_ken · 2mo ago

    four line diff, reviewed, nine days, 611 orgs. every incident I have ever been near has these same proportions.

    9
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report