Ask
101
@row_level_sam ·

msw or a real neon branch per PR when the suite has 9 minutes of budget CI

400 tests, nine minutes for the whole PR check including build and typecheck. Right now everything is mocked, including our own Postgres layer, which I know is silly and which I inherited.

Options: keep mocking, or spin a Neon branch per PR and run the repository tests against real Postgres.

The thing I actually care about is RLS policies being correct. That is the failure that leaks another tenant's data, and it is the one I have no coverage of at all.

8 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @merino_max · 6d ago

    The real cost of request mocking is drift. That mocked payment provider response was accurate fourteen months ago. Nothing tells you when it stops being.

    If you keep mocks for third parties, generate the fixtures from a recorded real response and re-record on a schedule against a sandbox account. Otherwise you are testing your own memory of an API.

    78
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @r2_bucketeer · last wk.

    Test the policy from outside, as the app role. Set the role and the request claims exactly the way your platform does at runtime, then assert that selecting another tenant's rows returns zero rows.

    Assert on the row count, not on an error message. Error text changes between versions and you will spend a morning on it for no benefit.

    51
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @secondhandamps · last wk. · 2 replies

    Point them at the staging database. Branches are a lot of setup for the same outcome and you get realistic data for free.

    27
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @thermal_theo · last wk.

      Shared staging plus parallel PRs means two runs truncating each other's tables, and you will chase that flake for a month before you work out it was never your code. It also means any test anyone writes can now delete staging data, and someone eventually writes that test.

      Isolation is the entire point of the exercise. Realistic data is a separate problem, solved by seeding an anonymised snapshot into the branch.

      18
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @wick_and_wax · last wk.

    Budget maths, since nine minutes is the actual constraint: 400 tests against a real database is fine as long as they are not each running migrations. Migrate once in global setup, then isolate per test by tenant id, or truncate the three tables that matter rather than all forty.

    What blows a nine minute budget is not Postgres, it is running your migration chain 400 times and creating a connection per test.

    47
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @kombucha_kai · last wk.

    Add one canary test that only passes when RLS is actually on. Otherwise the day someone runs CI as a superuser or a role with bypass privileges, every policy test goes green and stays green, and the suite is worse than useless because it is now lying confidently.

    23
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @otis_haverford · last wk.

    In-process Postgres compiled to wasm is a third option worth knowing about - genuinely Postgres, so policies behave, and no container to wait for. A sqlite stand-in is not an option here at all, since it has no row level security to test.

    20
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report