every generated test mocks the thing it is supposed to test, 40 files in tests
Inherited a service with no tests. Asked for coverage, got 40 Vitest files, 380 tests, all green, 84% coverage. Felt great for about a day.
Then I deliberately broke a validation rule — changed a >= to a > in the middle of the pricing logic — and every test still passed.
Looking properly, most files mock the dependencies of the module under test so thoroughly that the assertions are checking the mocks. There is a test literally called calculates tax correctly that mocks calculateTax and asserts it was called once.
Is there a way to ask for tests that don't do this, or was 380 tests just the wrong ask?
@coworking_cass · 3mo ago · 3 replies
380 tests was the wrong ask. Coverage as a target produces exactly this — the cheapest way to execute a line is to mock everything around it, and you asked for lines executed.
Two changes fix it.
Ask for behaviour rather than coverage: "write tests for the pricing rules in this file, table-driven, real inputs and expected outputs, no mocks except the HTTP client." Naming what may be mocked is the important part; the default is to mock everything reachable.
Then verify with mutation testing rather than coverage. Stryker pointed at the pricing module will do automatically what you did by hand — flip a
>=, run the suite, report that nothing failed. Start with one module, it's slow. A mutation score is the only coverage-adjacent number that can't be gamed by mocking.Also: delete the 40 files. Seriously. They're 380 tests of assurance you do not have, and keeping them means every future failure is ambiguous.
Reply
Report
@drainfield_dez · 3mo ago
Ran Stryker on the pricing module overnight. 71% line coverage, 9% mutation score. Deleted the lot and rewrote 22 tests by hand with the model filling in table rows.
Reply
Report
@heirloom_hank · 3mo ago
9% is a great number to have. It turns 'these tests feel useless' into something you can put in front of a colleague.
Reply
Report