Ask
172
@drainfield_dez ·

every generated test mocks the thing it is supposed to test, 40 files in tests

Inherited a service with no tests. Asked for coverage, got 40 Vitest files, 380 tests, all green, 84% coverage. Felt great for about a day.

Then I deliberately broke a validation rule — changed a >= to a > in the middle of the pricing logic — and every test still passed.

Looking properly, most files mock the dependencies of the module under test so thoroughly that the assertions are checking the mocks. There is a test literally called calculates tax correctly that mocks calculateTax and asserts it was called once.

Is there a way to ask for tests that don't do this, or was 380 tests just the wrong ask?

10 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @coworking_cass · 3mo ago · 3 replies

    380 tests was the wrong ask. Coverage as a target produces exactly this — the cheapest way to execute a line is to mock everything around it, and you asked for lines executed.

    Two changes fix it.

    Ask for behaviour rather than coverage: "write tests for the pricing rules in this file, table-driven, real inputs and expected outputs, no mocks except the HTTP client." Naming what may be mocked is the important part; the default is to mock everything reachable.

    Then verify with mutation testing rather than coverage. Stryker pointed at the pricing module will do automatically what you did by hand — flip a >=, run the suite, report that nothing failed. Start with one module, it's slow. A mutation score is the only coverage-adjacent number that can't be gamed by mocking.

    Also: delete the 40 files. Seriously. They're 380 tests of assurance you do not have, and keeping them means every future failure is ambiguous.

    164
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @drainfield_dez · 3mo ago

      Ran Stryker on the pricing module overnight. 71% line coverage, 9% mutation score. Deleted the lot and rewrote 22 tests by hand with the model filling in table rows.

      43
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @heirloom_hank · 3mo ago

      9% is a great number to have. It turns 'these tests feel useless' into something you can put in front of a colleague.

      27
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @muslin_mira · 3mo ago · 2 replies

    The reliable pattern for me: write the first test myself, badly. One test, real inputs, the assertion I actually care about. Then hand it over with "add cases in this exact style, same structure, no new mocks."

    Given a concrete example it copies the shape faithfully. Given a blank file and a coverage target it produces mock theatre. The failure is almost entirely about which of those two situations you put it in.

    71
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @wonder_why_wren · 3mo ago

      This is the whole trick for most codegen honestly. One good example beats three paragraphs of instructions.

      19
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @negativesplitz · 3mo ago

    Make the bad pattern impossible rather than discouraged. A lint rule that bans vi.mock in files matching your rules/domain test glob means those tests physically cannot mock, so they have to compute.

    Doesn't help with the existing 40 files, but it stops the next 40.

    45
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @iris_tanaka · 3mo ago

    Your calculates tax correctly example is worth keeping as a teaching artefact. A test that mocks its subject and asserts on the mock is a tautology — it passes for every implementation, including no implementation at all. Not a weak test, a zero-information one.

    The concrete tell: if you can delete the function body and the test still passes, delete the test.

    29
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @mature_student_j · 3mo ago

    84% coverage on an inherited service with zero tests, in one go, was always going to be false. Real coverage on legacy code needs refactoring for testability first, and the model routed around that by mocking — which is exactly what a person under time pressure does too.

    18
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @stropping_sal · 3mo ago · 2 replies

    coverage is a bad metric, everyone already knows this

    17
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @coworking_cass · 3mo ago

      Coverage is a fine metric read as a floor — 12% means you definitely have a problem. It's useless as a target, which is the thing that happened here. Different claim.

      15
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report