Architecture
Three layers, clean boundaries. The engine doesn’t know about parallel execution. The LLM client doesn’t know about pipelines.
System layers
Each box: layer name · source path · what it owns.
cmd/tracker
flags · name resolution · TUI · embedded workflows"] Engine["Pipeline Engine
pipeline/
run loop · edge selection · checkpoints · budget guard · steering channel"] Handlers["Handlers
pipeline/handlers
codergen · tool · human · parallel · fan_in · conditional · subgraph · manager_loop"] Backends["Agent Backends
pipeline/backend.go
native · claude-code · acp"] Agent["Agent Session
agent/
turn loop · tool registry · context compaction · episodic memory · verify-after-edit"] LLM["LLM Client
llm/
Anthropic · OpenAI · Gemini · OpenAI-compat · retry · cost estimation"] CLI --> Engine --> Handlers --> Backends --> Agent --> LLM
The engine run loop
Every node is treated the same: execute the handler, evaluate edges, advance. Parallel, subgraphs, human gates — handlers encapsulate it all.
context"] --> B["Execute
handler"] B --> C["Set outcome
in context"] C --> D["Select
outgoing edge"] D --> E["Checkpoint"] E --> F["Advance to
next node"] F --> A
Edge selection priority
Suggested next nodes
If the handler set
suggested_next_nodes(e.g., parallel pointing at a fan-in), take that edge.Conditional match
Evaluate conditions in declaration order; first match wins. Operators:
=,!=,contains,not contains,startswith,endswith,in, plus logicaland/or/not.Label match
Match a human gate’s
preferred_labelagainst edge labels.Default edge
First unconditional edge. Safety net — and the source of infinite loops if pointed at a loop target.
Handler types
Each node resolves to a handler. The engine calls Execute() and reads the outcome.
New handlers register without touching the run loop.
| Handler | Node kind | What it does |
|---|---|---|
codergen | agent | Runs an LLM session via the selected backend. Emits an agent.Event stream. |
tool | tool | Shell command with timeout, output cap, env scoping, denylist/allowlist, comment stripping for LLM-written lists. |
wait.human | human | Choice / freeform / hybrid / yes-no / interview gates. Pluggable Interviewer for autopilot, auto-approve, webhook routing. |
parallel | parallel | Dispatches branches from parallel_targets as goroutines. Honors max_concurrency and branch_timeout. |
parallel.fan_in | fan_in | Joins parallel branches; aggregates branch SessionStats into its own trace entry. |
conditional | conditional | Pure routing — evaluates conditions without an LLM call. |
subgraph | subgraph | Runs a referenced child pipeline; events propagate to the parent TUI. |
stack.manager_loop | manager_loop | Async child-pipeline supervisor. Polls, evaluates stop and steer conditions, injects context mid-execution. Native .dip syntax via ir.NodeManagerLoop. |
The dippin adapter
pipeline/dippin_adapter.go bridges dippin-lang’s IR to tracker’s Graph model.
Every naming mismatch lives here.
| Dippin IR | Tracker Graph | Notes |
|---|---|---|
model | llm_model | Attribute rename |
provider | llm_provider | Attribute rename |
google | gemini | Provider name normalization |
ctx.outcome | outcome | Namespace prefix stripped at evaluation time |
ParallelConfig.Targets | Implicit edges | Synthesized from parallel targets |
FanInConfig.Sources | Implicit edges | Synthesized from fan-in sources |
ir.NodeManagerLoop | handler=stack.manager_loop | Flattens the IR config into six unprefixed attrs (subgraph_ref, poll_interval, max_cycles, stop_condition, steer_condition, steer_context). steer_context uses canonical k=v,k=v with percent-encoding. Steered values land under ${ctx.steer.<key>} on the child to avoid colliding with the tool_command safe-key allowlist. |
Agent backends
AgentBackend is a one-method contract: execute a node, return an agent.Event stream.
The engine doesn’t know which backend ran. Mix per node; per-node attr beats --backend.
Native
Wraps agent.Session — turn loop, tool registry, context compaction, optional plan-before-execute, episodic memory, structured reflection on tool failure. Anthropic, OpenAI, Gemini, OpenAI-compat.
Claude Code
Spawns claude as a subprocess; parses NDJSON. File editing, terminal, MCP servers come for free. API keys stripped by default so the CLI uses subscription auth (override with TRACKER_PASS_API_KEYS=1).
ACP
Spawns any Agent Client Protocol agent over stdio. Provider-based defaults: anthropic → claude-agent-acp, openai → codex-acp, gemini → gemini --acp. Override per node via acp_agent.
Lazy LLM client. buildLLMClient() failure is non-fatal when the global --backend isn’t native.
A pipeline can run with no API keys at all if every node uses claude-code (subscription auth) or acp (bridge native auth).
Agent-runtime tools
The native backend ships an extensible tool registry. Architect-tier tools register on demand
via env vars — no env var, no tool. Lives in agent/tools/; registered from
pipeline/handlers/backend_native.go.
TerminalTool — primitive
An interface a tool can implement to flag itself terminal. The runtime ends the session loop the moment it succeeds — before the next LLM call — so dispatch tools don’t waste a post-dispatch turn.
dispatch_sprints
Reads a {path, description} JSONL plan and runs the per-sprint author + audit pipeline once per line. Bounded retry+backoff for transient provider errors; non-retryable errors bubble out.
write_enriched_sprint
Mid-tier LLM (Sonnet by default) with a 4-strategy SEARCH/REPLACE matcher (exact → indent-preserving → whitespace-insensitive → fuzzy). Partial-apply semantics distinguish PATCHED-PARTIAL from clean PATCHED. Override via TRACKER_SPRINT_WRITER_MODEL.
generate_code
Cheap/fast model expands a contract into one or more files. Default gpt-4o-mini; override via TRACKER_CODEGEN_MODEL. Companion to write_enriched_sprint.
State, budgets & resume
The PipelineContext is the typed key-value store shared across nodes.
Variable expansion resolves ${ctx.*}, ${params.*}, ${graph.*},
and ${ctx.node.<id>.*} in prompts, tool commands, and edge conditions — single-pass,
never re-scanning resolved values.
Per-node scoping
After each node, every key it wrote is also copied into node.<nodeID>.<key>. Downstream reads ${ctx.node.ReadSpec.last_response} reference a specific upstream output instead of the latest.
Declarative writes: / reads:
Nodes declare the keys they produce and consume. Declared writes are validated into the context after the handler runs — missing required fields fail the node. Self-healing JSON cascade catches the common “LLM returned prose” failure modes.
Budget guard
Enforces max_total_tokens / max_cost_cents / max_wall_time between every pair of nodes. Breach halts the run with OutcomeBudgetExceeded and a full CostSnapshot. Subgraph and manager_loop children inherit the parent guard so ceilings bind through nested execution.
Cost-accounting fidelity
Per-call usage carries an Estimated flag that propagates through SessionStats → ProviderUsage → CostSnapshot. Estimated rollups render as ~$X.XXXX (estimated). Cache reads priced at 10% of input, cache writes at 25%.
Git artifacts
With WithGitArtifacts(true), the run dir is a git repo. Every terminal node creates a commit (node(<id>): <handler> outcome=<status>) carrying duration, edge, token, and cost metadata. git log is a human-readable audit trail.
Resume & bundle
tracker.Config.ResumeRunID resolves a run ID to its checkpoint.json path. ExportBundle wraps git bundle create --all into a single self-contained file you can git clone anywhere — no network needed.
.dipx bundle identity v0.26.0
Content-addressed .dipx bundles (from dippin pack) load via pipeline.LoadDipxBundle: SHA-256 verifies every file in manifest.json, uses the pre-parsed *ir.Workflow directly (no re-parse), bypasses the filesystem subgraph walker. The sha256:<hex> identity is stamped on every activity.jsonl line, persisted into checkpoint.json, surfaced in tracker list/audit, and exposed on tracker.Result.BundleIdentity & RunSummary.BundleIdentity. Resume strictly verifies identity match; --force-bundle-mismatch is the escape hatch.
Steering channel. pipeline.WithSteeringChan(chan map[string]string) lets a supervisor inject context updates between node executions
via MergeWithoutDirty — writes the global namespace without leaking into per-node scope.
Used by stack.manager_loop to steer a child pipeline mid-run.
Steered values are namespaced under steer. on the child.
Run terminal statuses
EngineResult.Status carries one of four values when a run ends. The enum is open —
future minor releases may add values, so consumers should classify via TerminalStatus.IsSuccess()
(or the status_class JSON companion field) rather than switching on the raw string.
| Value | Meaning | IsSuccess() |
|---|---|---|
success | Run reached the success exit; every validation along the way passed. | true |
validation_overridden v0.35.0 | Run reached the success exit, but a human, autopilot, or webhook accepted a failed validation along the way via a wait.human edge marked override: true. Routing is unchanged from the success path — the marker is an audit signal recorded on EngineResult.ValidationOverrides. | true |
budget_exceeded | The BudgetGuard halted the run after a token, cost, or wall-time ceiling tripped. CostSnapshot on the result names the dimension. | false |
fail | Run halted via failure (strict failure edge, provider error, etc.). | false |
The tracker run CLI maps fail/budget_exceeded to exit 1 and (with --fail-on-override) maps validation_overridden to exit 2. Without the flag, override runs exit 0 — the default treats an operator-accepted override as audit-positive. See the CLI exit-codes table.
LLM client middleware
Provider-agnostic. Middleware wraps every request for cross-cutting concerns.
502, 503, 429"] Retry --> Token["Token
tracker"] Token --> Provider["Provider
adapter"] Provider --> API["Provider API"] API --> SSE["SSE Stream"]
Infra retry handles transient errors transparently: max 3 retries, 2s base delay, exponential backoff.
Pipeline-level retries are reserved for actual node logic failures.
SSE error handling matters because OpenAI’s Responses API returns HTTP 200 and sends
error / response.failed as SSE events — the adapter parses these out;
they are NOT reflected in the HTTP status.
TUI model
Built on bubbletea. The pipeline runs in a
background goroutine; the TUI receives events via tea.Msg. No polling.
- Pipeline panel — node list in topological order (Kahn’s algorithm). Status lamps, thinking spinners, tool indicators. Subgraph nodes inserted and indented dynamically.
- Activity log — append-only, line-level styled. No glamour rendering (too expensive for streaming). Per-node streams with separators on node change.
- Modal overlays — choice radio, freeform textarea, hybrid radio+freeform, fullscreen review with glamour viewport. All
Cancellable. - Event flow — engine emits pipeline events → adapted to
tea.Msg→ sent to program → state store updates → view re-renders.
Library API
Tracker is also a Go library. The CLI is a thin printer over the same APIs — and one
transport among peers. The TUI, the Slack bot, and any future
web or mobile app plug into the same tracker.Config → Engine boundary.
Structured reports
tracker.Diagnose, DiagnoseMostRecent, Audit, Doctor, Simulate return typed reports. Typed enums for CheckStatus and SuggestionKind — no stdout scraping.
NDJSON streaming
tracker.NewNDJSONWriter(io.Writer) produces the same wire format as --json. Plug it into Config.EventHandler, Config.AgentEvents, and the LLM trace hook. Per-writer write-failure recovery.
Context-first
Every entry point takes context.Context as its first argument, honored by provider probes and binary version lookups. Cancellation is real.
Typed NodeConfig
*pipeline.Node exposes AgentConfig / ToolConfig / HumanConfig / ParallelConfig / RetryConfig. Graph-default-then-node-override merge happens once inside the accessor; handlers consume typed structs, not raw Node.Attrs reads.