Prove the Test Can Fail: Mutate the Fix

After you write the test that goes with your fix, revert the fix and watch the test fail. That takes about thirty seconds, and no other single habit has done as much for my confidence in a diff.

That’s it. That’s the whole technique. I now consider a regression test unverified until I’ve seen it red, and I ask for the same from anything that writes code on my behalf.

Why a green test proves less than you think

A test written after a fix, against the fixed code, has never once been asked the question it exists to answer. It asserts that the current behavior is the current behavior. That statement is true for a correct implementation and equally true for a subtly wrong one, and you cannot tell which you have by running it.

The mutation makes the test earn its keep. If reverting the fix leaves the suite green, then one of two things is true: the test does not exercise the changed code path, or it does exercise it and does not assert on the thing that changed. Both mean the test will not catch a regression, which is the only job it had.

The failure message is also the deliverable, not a side effect. When I revert a Kelly sizing fix and get

FAIL: four numeric parameter cases failed as expected

I have learned something narrow and useful. When I revert a bankroll change and see the ending balances go from [100.0, 100.0] to [100.0, 115.0], I now know exactly which behavior the test pins. Record that in the pull request and a reviewer can trust the test without reading it.

The procedure

1. Write the fix. Write the test. Confirm green.
2. Revert only the fix, leaving the test in place. git stash a hunk, or edit the one line back.
3. Run the focused test, not the whole suite. Confirm red, and read the message.
4. Restore the fix. Confirm green again, and confirm your working tree is clean.
5. Put the red message in the pull request description.

Step five is the part people skip and the part that compounds. A reviewer reading “mutation check: reverting the guard failed test_rejects_cross_origin with 200 instead of 403” does not have to reason about whether the test binds to the change. You already did.

Three shapes that survive a naive check

The procedure above catches the easy case. These three are where I’ve seen it get subtle.

A guard whose test passes without the guard. You add a validation check and a test that sends bad input and expects an error. Both green. But the error was already being raised downstream by something else, so removing your guard changes the traceback and not the outcome. The test passes either way. The tell is that reverting produces a different failure rather than no failure, so you have to read the message and not just the exit code.

A fix with two independently load-bearing halves. Some fixes are genuinely two changes: validate the input, then re-check the resolved result. Path handling is the canonical case. You check that a requested path is inside an approved directory, and you check that the resolved path is too, because a symlink inside an approved directory can point out of it. Revert half of that and the suite stays green, because the test that would catch the second half, a symlink pointing outside, was never written. Belt and braces looks like redundancy and is actually an untested branch. Mutate each half separately.

A reintroduced bug that doesn’t fail loudly. Remove a retry cap and the suite does not fail, it hangs, so you need an external timeout for the mutation to be observable at all. Drop an await in async code and the runner dies with an unhandled rejection instead of a named test failure. In both cases the mutation “worked” and the signal arrived in a channel you weren’t watching. If your mutation produces a hang or a crash rather than a failure, that still counts, but you have to look for it deliberately.

What this is not

This is not mutation testing in the automated sense, and the two complement each other. A tool like mutmut generates mutants across your whole codebase and tells you which ones your suite fails to kill, which is a coverage-of-assertions measurement and a good one. I have an MCP server for driving it precisely because I want that number available while I work.

What I’m describing is the manual, targeted version aimed at exactly one thing: the test you just wrote, against the fix you just made. It costs thirty seconds instead of a full mutation run, and it answers the specific question a reviewer has. Use the tool for the codebase and the habit for the diff.

Where it earns the most

Two situations where I now consider it mandatory.

The first is anything numeric. A test with a hardcoded expected value is the easiest place in software to freeze a wrong answer, because the number came from somewhere and nobody remembers where. If the number came from running the code, the test is a snapshot, not a check. Reverting the fix and watching the value move from 25.33 to 29.40 tells you the assertion is bound to the math.

The second is any code I did not write myself, which now includes a lot of it. When something else writes the fix and the test together, they are consistent with each other by construction, and consistency is not correctness. The mutation check is the cheapest available way to find out whether the pair actually disagrees with the bug.

The one-line version

If a test has never failed, you have not tested anything. You have written a very confident comment.