Mostly Done.
Contents

5 Generate the proof

Provenance is not verification.

Documentation, screenshots, benchmarks, and marketing claims drift away from the software that made them true. Agentic development accelerates the drift because it accelerates both code and prose. An agent will update a README with the same confidence it uses to update an implementation, even when the measurement behind the README is stale or misunderstood.

If a claim can be generated, do not type it.

5.1 Make the marketing a test

Consider a math typesetting engine whose gallery figures regenerate from the live engine on every push. Six generators run in CI and publish the results, so a broken gallery image means either the engine changed or the example no longer means what the text says. A public claim has become an integration check.

Watch how easily that claim inflates. The engine’s announcement essay claimed every documentation figure was regenerated. On inspection, the two most persuasive images, the wordmark and a side-by-side comparison against an established engine, were committed static assets. Each had a reproducible generation script, but nothing in the build regenerated or validated them. The website’s scoped sentence, every image below, was accurate for the gallery. The essay’s global sentence was false, and the gap fell exactly where the claim carried the most weight.

The distinction is operational. A generated artifact has provenance: the system it advertises really did produce it once. A verified artifact is regenerated or checked by the current build. The wordmark has provenance. The gallery has verification. Only the second kind can catch a regression.

The side-by-side comparison deserves the same narrowing. It renders the same input through both engines at matched height, with no automated diff and no acceptable-difference tolerance. At equal height, one example’s aspect ratio measures 3.64 in the reference engine and 2.94 in the product: visibly close, not identical. The site’s phrase Judge the fidelity yourself is honest, because visual judgment is exactly what a side-by-side supports. It is not an equivalence test. Name the evidence you have.

Now consider a language toolchain whose machine-consumed specification is assembled by a script and embedded in the binary. A freshness test regenerates the spec into a temporary copy and fails if the checked-in file is stale, and every example in the repository is parsed by the real parser and linter in tests. That artifact does not merely have provenance. The build proves it is current on every run.

The technique generalizes:

The point is to eliminate hand-copied representations of facts the repository can derive.

5.2 Believe the measurement over the README

Consider a renderer whose README describes a cold render of about 0.3 ms covering parse through rasterization. A phase profiler, built to check that every microsecond of the path is understood, contradicts the claim. On macOS, the measured call returns a lazy image; rasterization happens later, on first paint. The number also came from a debug build, which the profiler shows runs the renderer’s own phases two to five times slower than release. The corrected figures separate what the old claim merged: a warm render around 0.5 microseconds, a cold fill around 0.2 ms, the deferred paint around 0.23 ms, and a headless render around 8 microseconds.

The corrected numbers matter less than the corrected model. Fix the README, commit the profiler, and the next performance claim can be regenerated instead of remembered.

The pattern is worth keeping:

  1. state the question in measurable phases;
  2. build the smallest instrument that separates them;
  3. run it in the environments whose behavior may differ;
  4. update the claim to match the measurement;
  5. keep the instrument close enough that the claim can be re-derived.

5.3 Generated does not mean trusted

Generation removes one class of drift and introduces another possibility: the generator can be wrong.

A screenshot pipeline can mirror an image, crop the wrong region, or capture a lazy placeholder. A benchmark can omit the expensive deferred work. A compatibility script can mistake workflow completed for artifact published. A documentation generator can preserve stale metadata faithfully.

So damage the proof system on purpose and confirm it objects:

Generated proof beats a typed assertion only when its source and verifier are inspectable.

5.4 Documentation should lose arguments with reality

It is tempting to treat documentation as a promise the code must be made to satisfy, even when the promise was written from a mistaken understanding. That temptation produces contorted implementations built to preserve a flattering sentence.

When a measurement disagrees with your README, fix the README unless the product contract truly requires a code change. When a design review proves a shipped headline feature semantically wrong, revert the feature rather than defend the announcement. When your live site is three releases behind, the work is not done, however long ago the merge happened.

Agents and people read documentation and act on it, so a wrong sentence propagates into wrong work. That makes documentation worth maintaining and still not sovereign. It should hold the best current model of the system and provide executable paths back to the facts.

5.5 Proof at each public boundary

A mature agentic project benefits from a proof chain:

source -> tests -> built artifact -> generated docs -> release -> live surface

Each arrow should have a check:

This is why merged is not shipped. The chain ends where the user meets the product.

Treat a public claim like an API: versioned, tested where you can, revised when the evidence changes.

use ← and → to turn pages