Ask
26

Comparing coding assistants after the limits moved: what actually differs

Went shopping when my usage stopped fitting. Most comparisons are about the model, and after trying several the model was the smallest difference.

What actually differs

  • What it decides to read. The biggest gap by far. Two tools on the same model give very different answers because one found the right three files and the other sent a directory. This is the product; the model is a component.
  • How it edits. Whole-file rewrites versus targeted patches. Rewrites are more likely to succeed and much more likely to quietly change something you did not ask about.
  • Whether it can run things. Running the tests and reading the failure is the difference between a suggestion and a fix.
  • What happens when it is unsure. The good ones stop and ask. The rest guess confidently, which is worse than stopping.

What barely differs

  • Raw quality on a single self-contained function. All of them are fine.
  • The chat interface.

Practical
Test with your own repository, not a demo. Give each one the same real task, ideally one you have already done, and watch which files it opens before it writes anything. That single observation predicted my satisfaction better than any benchmark.

3 answers Share
Report

Answering anonymously, a moderator will review it first.

  • @harness_over_model · 3w ago

    The retrieval point is right and it explains why the same model feels different in two tools. What gets sent is the product and it is invisible, so people attribute the difference to the model.

    If a tool lets you see what it sent, look once. Mine was including a lock file and three build artefacts in every request, which was eating the budget and burying the actual code.

    24
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @local_for_routine · 3w ago

    The split that worked for me: routine work on something cheap or local, hard work on the expensive one. Renaming, tests, boilerplate and commit messages are most of the volume and none of it needs the good model.

    Deciding rule is how many files the answer depends on. One file, cheap. Several, expensive. That has held up for about six months.

    18
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @assistant_limits · 3w ago

    The "what happens when it is unsure" one is worth testing deliberately: give it a task with a genuine ambiguity and see whether it asks or guesses. It is the fastest signal I found and no benchmark covers it.

    1
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report