How we verify tracker

Golden-trace conformance

The tracker-conformance binary is the executable snapshot of engine behavior. It runs an example workflow through the real engine with a stub completer — no API keys, no wall-clock or run-id noise — and emits a normalized, deterministic trace: the event sequence, per-node SessionStats, the aggregate UsageSummary, and the terminal status and class.

The committed golden files (cmd/tracker-conformance/testdata/golden/*.golden.json) are versioned in lockstep with the tracker tag. That lockstep atomicity is the entire point: the goldens are regenerated in the same commit as the intentional engine change that altered them, and the binary ships as a release asset. A downstream port pins a tracker version, runs tracker-conformance golden <fixture.dip>, and diffs the output against its own expectation. Any event-schema, handler-contract, or usage-shape change surfaces as a diff instead of drifting silently past tests written against an old shape.

This is why the harness lives in the tracker repo and imports the engine's internal packages by design. Splitting it out would break the one property that makes it valuable — there is no version of tracker-conformance that is meaningfully separable from the engine tag it pins.

# Print the normalized golden trace for a fixture
tracker-conformance golden validation_overridden.dip

# Regenerate goldens after an intentional engine change
go test ./cmd/tracker-conformance -run TestGoldenTraces -update-golden

Full contract, coverage list, and the schema-v2 normalization rules: embedding.md §5.

Complexity ratchet

A pre-commit and CI gate (make complexity) that grandfathers a baseline of existing cyclomatic / cognitive / file-size numbers and lets it only shrink. A new or worse violation fails the build; the baseline is burned down deliberately with make complexity-update.

It is not a style preference — it is a one-way ratchet that stops the engine from accreting the kind of untested complexity that hides behavior drift.

Docs-drift gate

A mechanical check (make docs-check, scripts/docs/gate.sh) that keeps the docs structurally in sync with the code. Every tracker subcommand must be documented in the CLI reference before it can ship, and a release's version must appear in both CHANGELOG.md and the website changelog before the tag can publish.

It is a structural check only — it does not judge prose completeness, which stays a human review responsibility. But it makes the specific "we shipped a command nobody documented" and "we tagged a release with a stale website" failures impossible.

Provider round-trip conformance

A CI gate that mechanically catches the "omitempty drops a required-but-empty field on replay" bug class across the Anthropic, OpenAI, and Google adapters — the failure mode that produced real provider 400s (#567/#568).

Two complementary checks: a dynamic replay-fidelity test per provider parses a response block, re-serializes it as a request through the real translation path (including the stream-accumulator path), and asserts every schema-required key survives; and a static structural guard reflects over the request structs and flags any "empty-is-a-legitimate-value" required field left as a droppable omitempty. A lossy translation struct fails go test before any model hits it in production.

SWE-bench (coming)

The tracker-swebench harness (documented in the CLI reference) evaluates tracker's agent against the industry-standard SWE-bench dataset. The first scored, reproducible run will be published here.

No score is claimed until that run lands — a harness with no published number is machinery, not evidence. This slot is intentionally reserved.