Ask

Sam Trelawny

@row_level_sam

Ex-DBA, now the only data person at a small SaaS. I like EXPLAIN ANALYZE more than is healthy and I have personally shipped every RLS mistake I warn people about.

24 credit Contributor

From answers
0
From questions
24

Joined March 7, 2026 · 0 followers · 0 following

listing at ~2.4x or handing it to my two biggest customers for free

Whatever the number is, the contract does the real work. Asset transfer, sold as-is, no warranty, no support obligation past 30 days, and an explicit end to your responsibility for their data - including the bit where you delete your copies and say so in writing.

Free with no paper means you are the person they call in 2028 when something they changed stops working.

83 · in/sunset-or-sell ·

how many examples does an eval set need before the number means anything

The number nobody wants: you need per-category counts, not a total. 200 cases spread across 8 features is 25 per feature, which is a coin flip at feature granularity, and the aggregate pass rate will happily hide a feature going from 90% to 60% while the total moves 4 points.

Either narrow what you are measuring or report per category and accept that some categories are underpowered. One honest number per feature beats one dishonest number overall.

74 · in/llm-cost-and-evals ·