AskMeAnything
Ask
Ask
78
@yaml_yuri ·

build the eval harness in-house or pay for a platform with two engineers observability

Two engineers, about 200 daily active users, one LLM feature that matters. We currently ship prompt changes by reading twenty outputs and feeling reasonably good about them, which is obviously not sustainable.

We need regression testing on prompt changes plus trace visibility when something goes wrong. Vendors are in the $100-200 a month range. Building a runner is maybe a weekend. Where do people land on this in practice?

9 answers Share
Report

9 answers

BestTopNewOldControversial

Answering anonymously — a moderator will review it first.

  • @braise_or_bust · 5mo ago · 3 replies
    @braise_or_bust ·

    Build the eval set, buy the harness.

    The part that is valuable and non-transferable is your 150 labelled examples and the rubric you argued your way to. The runner, the trace viewer, the diff view and the labelling UI are commodity, and a weekend is an honest estimate for version one and a wild underestimate for the version that a non-engineer can use in month four.

    The split that keeps you portable: eval cases live in your repository as plain files, versioned and diffable alongside the prompt they test, and CI runs them. The vendor gets traces and the labelling interface. Then changing vendors costs you a day rather than a quarter, and you never have your ground truth locked inside someone else's database.

    57
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @yaml_yuri · 5mo ago
      @yaml_yuri OP ·

      Cases in the repo next to the prompt is the detail I would have got wrong. I was about to let the vendor own the test cases because their UI was nicer.

      19
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @vesting_cliff · 5mo ago
      @vesting_cliff ·

      That rule saved us when we switched platforms last year. The cases moved in an afternoon. Everything else we had built inside the old tool was throwaway and we knew it going in, which made the decision easy.

      15
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @pomodoro_pat · 5mo ago · 2 replies
    @pomodoro_pat ·

    At 200 daily actives your bottleneck is not tooling, it is that you have no labelled data. Spend the weekend labelling 100 real traces instead of building or buying anything, and you will know more about your product than either purchase would have told you.

    The harness can genuinely be a test file to start with. Run the cases, assert the pass rate, fail the build if it drops.

    42
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @visa_run_val · 5mo ago
      @visa_run_val ·

      Vitest as an eval runner is completely fine and we ran it that way for over a year. Cases in JSON, one test per case, a summary reporter that prints the pass rate, and a threshold that fails CI on a drop of more than two points. The only thing we ever missed was a decent way to look at a diff between two runs.

      18
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @harlan_voss · 5mo ago
    @harlan_voss ·

    Check what is in your traces before you pick either option. If they contain customer content, shipping them to a third party is a conversation with someone whose job title includes the word compliance, and that conversation takes longer than the build. Several of these tools have a self-hostable version, which is frequently the actual answer for teams in that position.

    22
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @vesting_cliff · 5mo ago
    @vesting_cliff ·

    The cost of building is not the $200 a month you save, it is that in-house tools get orphaned. If nobody owns it, three months from now you have a script nobody runs, an eval set nobody updates, and a prompt nobody dares change because the safety net rotted quietly. Buying puts a small recurring cost against a thing that keeps working when everyone is busy.

    31
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @seasonalpalette · 5mo ago · 2 replies
    @seasonalpalette ·

    We built it and it was fine. It also took four weekends rather than one, which is the only part of the original estimate I would correct.

    11
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @yaml_yuri · 5mo ago
      @yaml_yuri OP ·

      Four weekends against $200 a month puts the break-even somewhere around never, when I put it like that.

      9
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report