Architecture

Three layers, clean boundaries. The engine doesn’t know about parallel execution. The LLM client doesn’t know about pipelines.

System layers

Each box: layer name · source path · what it owns.

graph TB CLI["CLI
cmd/tracker
flags · name resolution · TUI · embedded workflows"] Engine["Pipeline Engine
pipeline/
run loop · edge selection · checkpoints · budget guard · steering channel"] Handlers["Handlers
pipeline/handlers
codergen · tool · human · parallel · fan_in · conditional · subgraph · manager_loop"] Backends["Agent Backends
pipeline/backend.go
native · claude-code · acp"] Agent["Agent Session
agent/
turn loop · tool registry · context compaction · episodic memory · verify-after-edit"] LLM["LLM Client
llm/
Anthropic · OpenAI · Gemini · OpenAI-compat · retry · cost estimation"] CLI --> Engine --> Handlers --> Backends --> Agent --> LLM

The engine run loop

Every node is treated the same: execute the handler, evaluate edges, advance. Parallel, subgraphs, human gates — handlers encapsulate it all.

graph LR A["Clear routing
context"] --> B["Execute
handler"] B --> C["Set outcome
in context"] C --> D["Select
outgoing edge"] D --> E["Checkpoint"] E --> F["Advance to
next node"] F --> A

Edge selection priority

  1. Suggested next nodes

    If the handler set suggested_next_nodes (e.g., parallel pointing at a fan-in), take that edge.

  2. Conditional match

    Evaluate conditions in declaration order; first match wins. Operators: =, !=, <, <=, >, >= (numeric), contains, not contains, startswith, endswith, in, matches (regex), plus logical and/or/not v0.55.0.

  3. Label match

    Match a human gate’s preferred_label against edge labels.

  4. Default edge

    First unconditional edge. Safety net — and the source of infinite loops if pointed at a loop target.

Handler types

Each node resolves to a handler. The engine calls Execute() and reads the outcome. New handlers register without touching the run loop.

HandlerNode kindWhat it does
codergenagentRuns an LLM session via the selected backend. Emits an agent.Event stream.
tooltoolShell command with timeout, output cap, env scoping, denylist/allowlist, comment stripping for LLM-written lists.
wait.humanhumanChoice / freeform / hybrid / yes-no / interview gates. Pluggable Interviewer for autopilot, auto-approve, webhook routing.
parallelparallelDispatches branches from parallel_targets as goroutines. Honors max_concurrency and branch_timeout.
parallel.fan_infan_inJoins parallel branches; aggregates branch SessionStats into its own trace entry.
conditionalconditionalPure routing — evaluates conditions without an LLM call.
subgraphsubgraphRuns a referenced child pipeline; events propagate to the parent TUI.
stack.manager_loopmanager_loopAsync child-pipeline supervisor. Polls, evaluates stop and steer conditions, injects context mid-execution. Native .dip syntax via ir.NodeManagerLoop.

The dippin adapter

pipeline/dippin_adapter.go bridges dippin-lang’s IR to tracker’s Graph model. Every naming mismatch lives here.

Dippin IRTracker GraphNotes
modelllm_modelAttribute rename
providerllm_providerAttribute rename
googlegeminiProvider name normalization
ctx.outcomeoutcomeNamespace prefix stripped at evaluation time
ParallelConfig.TargetsImplicit edgesSynthesized from parallel targets
FanInConfig.SourcesImplicit edgesSynthesized from fan-in sources
ir.NodeManagerLoophandler=stack.manager_loopFlattens the IR config into six unprefixed attrs (subgraph_ref, poll_interval, max_cycles, stop_condition, steer_condition, steer_context). steer_context uses canonical k=v,k=v with percent-encoding. Steered values land under ${ctx.steer.<key>} on the child to avoid colliding with the tool_command safe-key allowlist.

Agent backends

AgentBackend is a one-method contract: execute a node, return an agent.Event stream. The engine doesn’t know which backend ran. Mix per node; per-node attr beats --backend.

graph TB CH["CodergenHandler"] --> SB["selectBackend()"] SB -->|"default"| NB["Native\nagent.Session\nturn loop + tool registry"] SB -->|"backend: claude-code"| CC["Claude Code\nclaude CLI subprocess\nNDJSON stdout"] SB -->|"backend: acp"| AC["ACP\nJSON-RPC over stdio\nclaude-agent-acp / codex-acp / gemini --acp"] NB --> Events["agent.Event stream"] CC --> Events AC --> Events

Native

Wraps agent.Session — turn loop, tool registry, context compaction, optional plan-before-execute, episodic memory, structured reflection on tool failure. Anthropic, OpenAI, Gemini, OpenAI-compat.

Claude Code

Spawns claude as a subprocess; parses NDJSON. File editing, terminal, MCP servers come for free. API keys stripped by default so the CLI uses subscription auth (override with TRACKER_PASS_API_KEYS=1).

ACP

Spawns any Agent Client Protocol agent over stdio. Provider-based defaults: anthropic → claude-agent-acp, openai → codex-acp, gemini → gemini --acp. Override per node via acp_agent.

Lazy LLM client. buildLLMClient() failure is non-fatal when the global --backend isn’t native. A pipeline can run with no API keys at all if every node uses claude-code (subscription auth) or acp (bridge native auth).

Agent-runtime tools

The native backend ships an extensible tool registry. Architect-tier tools register on demand via env vars — no env var, no tool. Lives in agent/tools/; registered from pipeline/handlers/backend_native.go.

TerminalTool — primitive

An interface a tool can implement to flag itself terminal. The runtime ends the session loop the moment it succeeds — before the next LLM call — so dispatch tools don’t waste a post-dispatch turn.

dispatch_sprints

Reads a {path, description} JSONL plan and runs the per-sprint author + audit pipeline once per line. Bounded retry+backoff for transient provider errors; non-retryable errors bubble out.

write_enriched_sprint

Mid-tier LLM (Sonnet by default) with a 4-strategy SEARCH/REPLACE matcher (exact → indent-preserving → whitespace-insensitive → fuzzy). Partial-apply semantics distinguish PATCHED-PARTIAL from clean PATCHED. Override via TRACKER_SPRINT_WRITER_MODEL.

generate_code

Cheap/fast model expands a contract into one or more files. Default gpt-4o-mini; override via TRACKER_CODEGEN_MODEL. Companion to write_enriched_sprint.

Filesystem jail. A native-backend agent node can carry writable_paths to bound where it may write. On Linux this re-execs tracker under a Landlock ABI v3 sandbox before running; in-process tools (Write, Edit, ApplyPatch) additionally resolve every path via openat2(RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS | RESOLVE_NO_MAGICLINKS) against a session-root fd. The gate refuses to start on a malformed glob, an acp/claude-code backend, or a kernel without Landlock v3 — it never silently runs unconfined. Residual escape classes (network egress, reads/exfil, anything inside an allowed path) are documented, not bounded — see the Linux security primitives and writable-paths audit checklist v0.53.0.

State, budgets & resume

The PipelineContext is the typed key-value store shared across nodes. Variable expansion resolves ${ctx.*}, ${params.*}, ${graph.*}, ${inputs.*}, and ${ctx.node.<id>.*} in prompts, tool commands, and edge conditions — single-pass, never re-scanning resolved values.

${inputs.*} is the workflow’s declared, caller-supplied run signature (a dippin inputs block; requires dippin ≥ v0.51). Values are introspectable (tracker.DescribeInputs), validated standalone with structured per-input errors (tracker.ValidateInputs) and bound at run start (Config.Inputs; a missing required input fails closed before any node runs), and untrusted by construction — the whole namespace is blocked from tool_command interpolation. file AND secret inputs are staged to a fixed 0600 path (.tracker/inputs/<name>) the workflow’s shell reads directly; for a secret, ${inputs.<name>} is that path only, so the value never enters a prompt, the provider wire, the trace, or the checkpoint v0.60.0.

Per-node scoping

After each node, every key it wrote is also copied into node.<nodeID>.<key>. Downstream reads ${ctx.node.ReadSpec.last_response} reference a specific upstream output instead of the latest.

Declarative writes: / reads:

Nodes declare the keys they produce and consume. Declared writes are validated into the context after the handler runs — missing required fields fail the node. Self-healing JSON cascade catches the common “LLM returned prose” failure modes.

Budget guard

Enforces max_total_tokens / max_cost_cents / max_wall_time between every pair of nodes. Breach halts the run with OutcomeBudgetExceeded and a full CostSnapshot. Subgraph and manager_loop children inherit the parent guard so ceilings bind through nested execution.

Per-node ceilings

Beyond the run-wide budget, a single node can carry max_cost_usd (a dollar cap on that node's own spend, firing EventNodeCostLimitExceeded) and no_progress_turns (a runaway detector that stops an agent whose turns stop advancing the workspace, firing EventNodeNoProgressDetected). Both are opt-in per node v0.52.0. The no-progress detector keys on a successful edit landing once an agent has shown it edits, so a tight-loop that re-reads files or re-runs a failing command every turn trips it v0.54.0.

Cost-accounting fidelity

Per-call usage carries an Estimated flag that propagates through SessionStatsProviderUsageCostSnapshot. Estimated rollups render as ~$X.XXXX (estimated). Base model prices come from dippin-lang/pricing — the single source of truth, so dippin cost estimates and tracker’s actual-spend math use identical numbers with nothing to drift v0.59.0. tracker keeps only per-model cache multipliers as an overlay (no phantom cache-write premium on OpenAI/Gemini v0.50.0) until dippin carries cache rates, and TokenTracker prices per (provider, model) so a two-model run is order-independent v0.51.0. A model dippin doesn’t price is no longer silently $0: run.json carries an unpriced signal and a --max-cost-can't-bound-this warning names the model — a signal, not a hard failure, so a genuinely-free local model still runs v0.52.0.

Run capture v0.50.0

A finished run is reconstructable: the executed spec (workflow.dip + workflow.ir.json + inputs), the verbatim provider request bodies, and per-call/turn/session identity land on every activity.jsonl line, rolled into a run.json manifest at run end. tracker run-json backfills one for archived or SIGKILLed runs. Capture files are written 0600 with O_NOFOLLOW, and .tracker/ never travels in exported bundles.

Git artifacts

With WithGitArtifacts(true), the run dir is a git repo. Every terminal node creates a commit (node(<id>): <handler> outcome=<status>) carrying duration, edge, token, and cost metadata. git log is a human-readable audit trail.

Resume & bundle

tracker.Config.ResumeRunID resolves a run ID to its checkpoint.json path. ExportBundle wraps git bundle create --all into a single self-contained file you can git clone anywhere — no network needed.

.dipx bundle identity v0.26.0

Content-addressed .dipx bundles (from dippin pack) load via pipeline.LoadDipxBundle: SHA-256 verifies every file in manifest.json, uses the pre-parsed *ir.Workflow directly (no re-parse), bypasses the filesystem subgraph walker. The sha256:<hex> identity is stamped on every activity.jsonl line, persisted into checkpoint.json, surfaced in tracker list/audit, and exposed on tracker.Result.BundleIdentity & RunSummary.BundleIdentity. Resume strictly verifies identity match; --force-bundle-mismatch is the escape hatch.

Steering channel. pipeline.WithSteeringChan(chan map[string]string) lets a supervisor inject context updates between node executions via MergeWithoutDirty — writes the global namespace without leaking into per-node scope. Used by stack.manager_loop to steer a child pipeline mid-run. Steered values are namespaced under steer. on the child.

Run terminal statuses

EngineResult.Status carries one of five values when a run ends. The enum is open — future minor releases may add values, so consumers should classify via TerminalStatus.IsSuccess() (or the status_class JSON companion field) rather than switching on the raw string.

ValueMeaningIsSuccess()
successRun reached the success exit; every validation along the way passed.true
validation_overridden v0.35.0Run reached the success exit, but a human, autopilot, or webhook accepted a failed validation along the way via a wait.human edge marked override: true. Routing is unchanged from the success path — the marker is an audit signal recorded on EngineResult.ValidationOverrides.true
budget_exceededThe BudgetGuard halted the run after a token, cost, or wall-time ceiling tripped. CostSnapshot on the result names the dimension.false
paused_billing v0.47.0Billing/quota exhaustion halted the run in a recoverable, resumable terminal (checkpoint + preserved WIP + tracker -r resume) rather than a fatal abort. Surfaced to embedders as RunManager.RunPaused, and classed paused (not failed) by tracker audit.false
failRun halted via failure (strict failure edge, provider error, etc.).false

The tracker run CLI maps fail/budget_exceeded to exit 1 and (with --fail-on-override) maps validation_overridden to exit 2. Without the flag, override runs exit 0 — the default treats an operator-accepted override as audit-positive. See the CLI exit-codes table.

LLM client middleware

Provider-agnostic. Middleware wraps every request for cross-cutting concerns.

graph LR Req["Request"] --> Retry["Retry
502, 503, 429"] Retry --> Token["Token
tracker"] Token --> Provider["Provider
adapter"] Provider --> API["Provider API"] API --> SSE["SSE Stream"]

Infra retry handles transient errors transparently: max 3 retries, 2s base delay, exponential backoff. Pipeline-level retries are reserved for actual node logic failures. SSE error handling matters because OpenAI’s Responses API returns HTTP 200 and sends error / response.failed as SSE events — the adapter parses these out; they are NOT reflected in the HTTP status.

TUI model

Built on bubbletea. The pipeline runs in a background goroutine; the TUI receives events via tea.Msg. No polling.

Library API

Tracker is also a Go library. The CLI is a thin printer over the same APIs — and one transport among peers. The TUI, the Slack bot, and any future web or mobile app plug into the same tracker.Config → Engine boundary.

Structured reports

tracker.Diagnose, DiagnoseMostRecent, Audit, Doctor, Simulate return typed reports. Typed enums for CheckStatus and SuggestionKind — no stdout scraping.

NDJSON streaming

tracker.NewNDJSONWriter(io.Writer) produces the same wire format as --json. Plug it into Config.EventHandler, Config.AgentEvents, and the LLM trace hook. Per-writer write-failure recovery. v0.47.0 The StreamEvent wire and the ActivityEntry reader are at full payload parity with activity.jsonl — cost rollups, edge decisions, gate lifecycle, and per-turn usage all decode losslessly, held to one field-name spelling by a mechanical parity test.

Run capture seam v0.51.0

Setting Config.Capture wires the same on-disk run capture the CLI produces — activity-log capture, trace observer, session-owned de-dup, and run.json finalization — from one field, so a library caller and the CLI share one path and can’t drift.

Policy & diagnostics seams

SessionConfig.Guardrail gates every agent tool call on (tool, args, context) before it runs, fail-closed v0.48.0. tracker.SetDiagnosticLogger routes library diagnostics through an injectable log/slog sink (no-op by default) instead of the global logger.

Context-first

Every entry point takes context.Context as its first argument, honored by provider probes and binary version lookups. Cancellation is real.

Typed NodeConfig

*pipeline.Node exposes AgentConfig / ToolConfig / HumanConfig / ParallelConfig / RetryConfig. Graph-default-then-node-override merge happens once inside the accessor; handlers consume typed structs, not raw Node.Attrs reads.