Mostly Done.
Contents

4 A test that has never failed is a rumor

A green test is evidence only if it can turn red for the defect it claims to guard. That sounds obvious, and it is among the most commonly skipped steps in agentic work, because an agent often generates the implementation and the test in the same pass. The two can share one misunderstanding. They agree beautifully and are wrong together.

4.1 Revert the fix

The simplest discipline is the strongest:

  1. reproduce the bug;
  2. write the regression test;
  3. apply the fix;
  4. observe the test pass;
  5. revert or disable the fix;
  6. observe the test fail for the expected reason;
  7. restore the fix and run the relevant suite again.

The red run can come from either direction: stash the fix and watch the new test fail, or run the test against the buggy code before the fix lands. Consider a leak fix in file-watcher teardown. Run against the buggy code, the regression test created 150 watchers and caught the process holding 153 descriptors against a baseline of 23:

XCTAssertLessThanOrEqual failed: ("153") is greater than ("23")

That failure is what made the later green mean something.

One qualification. Demonstrations like this are usually manual: you stash the revert by hand, and the red run survives only as a note in a PR description. Unless CI enforces mutation checks, nothing proves the test can still fail today. Was shown to detect the defect once is weaker than is continuously shown to detect it. A recorded red run is evidence with a date on it.

The deliberate red run catches mistakes a green run hides:

4.2 Test the test harness with deliberate damage

Consider a golden-image harness for a math renderer, built to catch blank or wrong output by computing an ink signature over each rendered equation. To prove the harness worked, its author zeroed one equation’s golden image, exactly the corruption it existed to catch. The test passed.

The signature averaged darkness across cells. Thin glyph strokes diluted toward white, and quantization erased what remained. The harness built to catch broken render passes silently passed broken renders silently. After the repair, the same tamper drifted 18 of 256 cells, 7.03 percent against a 2 percent limit. The zeroed golden finally failed loudly.

A second flaw surfaced later: the signature read only the red channel, so a red glyph could register as blank while sitting visibly on the page. Independent review found the blind spot. The fix, measuring ink by the darkest channel instead of the red one, was checked from both sides: a colored render’s inked cell count rose from 39 to 48 while the uncolored controls did not move. The correction changed what it should have changed and nothing else.

The general rule:

Every new verifier should ship with a known-bad probe that it rejects.

The probe can stay in the suite as a permanent test of the verifier, or live once in the change record. Keeping it is usually better.

4.3 Prefer differential and property tests for parsers

Suppose your agent hands you an escape-aware condition parser along with a dozen passing examples. Do not stop there. Write a slow, simple reference decoder and brute-force tens of thousands of generated inputs against it:

for input in generated_inputs:
    assert optimized_decode(input) == reference_decode(input)

Differential testing pays off most on agent-generated parsing code. A parser packs compact logic over a large behavioral surface, and a model can produce a plausible state machine that handles every example in the prompt and fails on combinations nobody thought to write by hand.

The reference need not be production quality. It exists only for comparison. Property tests add round trips, preservation, idempotence, and monotonicity. The important move is generating disagreement opportunities beyond the author’s examples.

4.4 Watch for tests that prove their implementation

A subtle failure occurs when the test reconstructs expected behavior with the same algorithm as the code under test. If both compute a midpoint by subtracting arrow lengths incorrectly, the test proves consistency, not correctness.

Stronger oracles come from a different representation:

Independence is not binary, but every distinct representation reduces shared blind spots.

4.5 Failure quality matters

A test that fails noisily for many unrelated reasons is hard to trust. The deliberate red run should fail on the assertion connected to the defect, not because the fixture crashes or the build no longer compiles.

This is why reproduction comes before the fix. It establishes the observable symptom and the shortest path to it. The regression test should preserve that path.

A useful test report answers:

That record matters most when the author is an agent and you never watched the test being developed.

The common slogan says do not trust the model. It stops short. A test carries no authority from its author, human or agent, and none from months of green. Trust a check in proportion to the evidence that it detects what it claims to detect.

use ← and → to turn pages