Agent Harness Engineering

Part of the PersonalClaw research-learnings library. Source-agnostic; distilled 2026-07-13 from a 95-source competitive-research corpus.

The harness is everything outside model weights that shapes agent behavior: system prompts and instruction files, tools and their descriptions, the execution environment and sandbox, the turn loop itself, context lifecycle, permission gates, verification feedback, and state management. Core equation: coding agent = model + harness, and observed behavior is dominated by the harness, not the model.

Principles (the durable truths)

Capability ≠ reliability; the harness is the lever. A controlled experiment with a fixed frontier model: bare run finished in 20 min for $9 and produced a broken app; the same model inside a planner/generator/evaluator harness ran 6 h for $200 and produced a fully working one. A separate open experiment held the model frozen and improved only the harness file: benchmark score went 0.560 → 0.780 (~40% relative) across 96 autonomous iterations. A decent model with a great harness beats a great model with a bad harness. Multiple independent systems converge on this.

Agent failures are legible configuration problems, not model problems. Smarter models will not end failures — they receive harder problems and fail in new ways. The discipline is maximizing today’s models. One team took the same model from 20% → 60% → 80% → ~100% task success purely by adding instructions, then executable verification commands, then progress files — no model change.

The ratchet. Add a constraint ONLY after a real observed failure (“ratchet, don’t brainstorm”); every instruction line must be traceable to a specific thing that went wrong; remove a constraint only when a more capable model makes it demonstrably redundant. Corollary: the right harness cannot be downloaded as a framework — it is shaped by your failure history.

Every harness component encodes an assumption about what the model can’t do alone. When the model improves, the component becomes load-bearing for nothing and should come out — but harnesses don’t shrink, they move: retiring old scaffolding raises the ceiling, which exposes new failure modes needing new scaffolding (multiday memory, multi-agent coordination, quality evaluators). Behavior-first design test: if you can’t name the behavior a component exists to deliver, it shouldn’t be there.

Deterministic rails beat probabilistic instruction (“march of nines”). Per-step reliability compounds: a 90%-per-step workflow fails too often to trust. If a step is mandatory, codify it in code, a state machine, a schema, or a hard gate — never merely ask the model to remember it. Production evidence: a >1,300-machine-PRs-per-week pipeline interleaves hard-coded gates (linter the agent cannot skip, deterministic commit step, deterministic context assembly before the model wakes) with LLM steps; “anything deterministic logic can solve never goes to a probabilistic model.”

Makers systematically overestimate completeness. Neural nets are overconfident by construction, and same-model self-evaluation is structurally generous (“a model is its own output’s best defense attorney”). Completion must be externally owned and verified — see the pass-state gating mechanism below and verification-and-judging.

The filesystem/repo is the system of record. An agent’s world = system prompt + repo files + tool output; anything else doesn’t exist. Proximity beats length (a 50-line ARCHITECTURE.md next to the code beats a 500-page wiki). The Fresh Session Test: a zero-history session must be able to answer what is this / how to run / how to verify / what’s unfinished / what’s next, from disk alone.

Blast radius scales with the number of turns a mistake survives. The same bug at each layer: prompt → one wrong answer; context → confidently wrong, then cleared; harness → one bad diff, caught in review; loop → written into a state file and read back next morning as established fact, load-bearing for days. Every harness mechanism (evaluators, checkpoints, budget caps) exists to shorten mistake-to-discovery distance.

Harness overfitting is real. Models are post-trained inside specific harnesses and perform differently elsewhere: one frontier model ranked ~#33 on a terminal benchmark in its native harness and ~#5 in a custom one; a team went Top-30 → Top-5 changing only the harness. The best harness is the one designed for your task, not the training harness.

Success is silent; failures are verbose. 4,000 lines of passing test output flooded context and caused hallucination; surface only errors, with full error text so the agent can self-correct.

Classify every control on two axes. Feedforward (guides: rules, docs, skills — steer before acting) vs feedback (sensors: tests, linters, judges, browsers — observe after, enable self-correction); and computational (deterministic, cheap, run on every change) vs inferential (LLM-judged, costly, run at gates). Feedback-only agents repeat mistakes; feedforward-only agents never find out whether they worked. Keep quality left: cheap computational checks pre-commit, expensive inferential checks post-integration, continuous sensors (drift detection) outside the change lifecycle.

Mechanisms (implementation-ready designs)

1. Agent loop anatomy

The canonical loop (converged on independently by several from-scratch rebuilds): while turns < max_turns (typical: 50–200 engine-level, ~8 per user input): pre-turn compaction check → stream model → on tool_use blocks, per-call pipeline (pre-tool hook → schema validation → normalized permission check → confirmation prompt if needed → execute → offload oversized output → record carryover state → post-tool hook) → all tool results appended as ONE user message (the API requires results in tool_use order). Parallel batching: coalesce consecutive concurrency-safe tool calls into one Promise.all batch (cap ~10) under return_exceptions=True semantics — one failing tool must not cancel siblings, because a request missing any tool_result is rejected. isConcurrencySafe is per-tool, optionally input-dependent, defaults false. Loop contract surfaces a typed event stream (text, thinking, tool_use start/done, permission_request, turn_complete{reason}, token_warning, api_retry, stream_restart, turn_usage) with termination reasons: completed / aborted / model_error / max_turns / blocking_limit.

2. Context compaction ladder (micro → full → reactive → breaker)

Four tiers, strictly ordered by cost:

• Micro-compact (structural, free): outside a keep-recent window (last ~8 messages), replace tool_results from a fixed compactable set (Read/Grep/Glob/Bash/Edit/Write, old MCP results) with "[Old tool result content cleared]"; collapse historical images to [image] markers. No LLM call.
• Full compact (LLM summarize): a 9-section summary prompt — primary intent / technical concepts / files+code / errors+fixes / problem-solving / ALL user messages / pending tasks / current work / next step — with a hard no-tools preamble (“tool calls will be REJECTED and waste your only turn”), then keep a verbatim tail. Tool-pairing tail invariant: walk the tail start backwards until no tool_result lacks its matching tool_use — a naive cut produces API-rejected dangling results. Applies identically to any transcript cut (rewind, fork, journal replay). Append a [CompactBoundary] type=… messages=… compacted_tool_ids=… marker so later logic can compute “messages since last compact” (enables incremental synthesis).
• Reactive compact (one-shot guard): on a prompt_too_long / context-length API error, force-compact at most ONCE per query (a reactive_compact_attempted flag), then retry the turn. Without the guard, compact→still-too-long→compact loops forever burning calls.
• Circuit breaker: N (=3) consecutive compaction failures disables auto-compact rather than thrashing. Token buffers (warning/auto/blocking thresholds) scale proportionally below a 180K reference window so small local models keep sensible ratios. Suppress auto-compact when the current call is the summarizer (no recursive compaction).

3. Tool-output offloading (exact mechanics)

Output beyond an inline threshold is written to data_dir/tool_artifacts/{timestamp}_{tool}_{12hexuuid} and replaced inline by a stub: truncation notice + tool name + tool_use id + original size + artifact path + a capped preview. Track the path as an “active artifact” the model can re-read on demand. At UI/journal boundaries, the same idea with a typed stub: >64KB inline base64 detected by magic prefix → {result_omitted: true, reason: "image_base64", bytes, path}. Head/tail-keep for large logs (keep the first and last chunk, offload the body). Multiple independent systems converge on offload-to-disk + stub + on-demand re-read.

4. Typed carryover buckets (task state that survives compaction)

Instead of trusting the summarizer with task state, maintain bounded dedup buckets fed by successful tool calls: tracked read files (cap 6; path + line-span + 6-line preview), tracked skills (8), work log (10), verified work (10), async agent tasks (12). These are re-injected across compressions — a typed side-channel of loop state, cheaper and more reliable than summarizer prose.

5. Full context resets with structured handoffs (reset > compact for long horizons)

Compaction keeps the “what” (code) and drops the “why” (rejected alternatives, constraints) — later sessions “optimize away” deliberate choices. For long tasks, compaction alone is demonstrably insufficient: tear down the session and rebuild from a compact handoff file (verified state / changes made / broken-unverified items / next action / commands to run). The Ralph-loop variant: a hook intercepts the model’s exit attempt and reinjects the original prompt into a fresh window; state persists via filesystem, turning a single-session agent into a multisession one (“agents work memoryless in shifts, relying on notes left on disk”). Rule of thumb: if a task needs >60% of the window, start preparing the handoff. Reset-vs-compact should be a per-model policy: “context anxiety” (rushing/skipping verification as context runs low) is strong in some model tiers and absent in others, so harness design is model-specific. Persist decisions as structured records {choice, reason, rejected_alternatives, constraints} — otherwise resumed sessions re-litigate settled decisions. Measured: a handoff-discipline case study cut rebuild cost 78% (target ≤3 min to executable state), lifted completion 58%→100%, dropped hidden defects 43%→8%.

6. Pass-state gating and engine-owned completion

The harness — never the agent — controls state transitions. Work items are triples (behavior, executable verification, state) with evidence; active → passing only via successful verification execution, and the transition is irreversible. A worker may only REQUEST a transition; the engine runs the declared verification and flips the state. Verified schema: {id, priority, area, title, user_visible_behavior, status, verification[], evidence[], notes} + rules {single_active_feature, passing_requires_evidence, do_not_skip_verification}; states not_started/in_progress/blocked/passing. Four harness components consume it: scheduler (picks next not_started), verifier (gates transitions), handoff reporter, progress tracker. Completion verification is a three-layer ladder — static analysis → runtime behavior → system-level e2e — strictly ordered, no skipping; and priority is correctness → performance → style with no refactoring before core functionality is verified. Measured effect of engine-owned completion in one case: real completion 37.5% → 87.5%. Track Verified Completion Rate (verified ÷ activated tasks) and block activating new work while VCR < 1.0. WIP=1 (“small next step”) yielded +37% completion, and LOC correlates negatively with features completed (800 lines/20% pass vs 200 lines/100% pass).

7. Session startup and clean-exit rituals

Fixed 9-step startup: confirm root → read progress file → read work-item list → git log --oneline -5 → run init script → run baseline smoke → if baseline broken, fix that first (never let new work obscure pre-existing breakage) → select highest-priority unfinished item → work only on it until verified or explicitly blocked. Mirror-image exit: log progress, update statuses, write handoff, commit, leave a clean restart path. Clean exit has five dimensions: build passes, tests pass, progress recorded machine-readably (cuts startup diagnosis 60–80%), no stale artifacts, standard startup path works. Session integrity is a transaction: commit clean or roll back. 12-week data: no cleanup strategy → 68% builds passing / 61% tests / 60+ min startup; with it → 97% / 95% / 9 min. Initialization is its own phase with its own objective (future-session reliability, not feature output): session 1 ships infrastructure only — runnable env, one verified passing test, task breakdown with acceptance criteria, baseline commit — gated by four conditions (can start / can test / can see progress / can pick next step). Dedicated init measured at +31% feature completion, recouped within 3–4 sessions.

8. Dual timeouts (idle vs completion)

Split agent-process timeouts into two phases keyed on the completion signal. Idle timeout (default 600s, reset on every output event) before any completion signal: genuinely stuck → fail. Completion timeout (60s silence-based grace window, reset per output line) after the signal is seen but the process won’t exit (a spawned child or MCP server holds stdout open so EOF never comes): resolve successfully with a warning, keeping collected commits/trailing token-usage events — instead of waiting 10 minutes and failing, discarding already-committed work. Rejected alternatives (documented): keying on a terminal stream event (none exists reliably across providers) and killing on signal (loses trailing data). The completion signal itself is a pure termination marker (default <promise>COMPLETE</promise>, string or array, first match stops) that the engine never injects into the prompt — caller-documented convention. 100-iteration implementer loops run on nothing more than max_iterations + completion signal(s) + dual timeouts.

9. Output-truncation recovery (two-phase)

On stop_reason=max_tokens: phase 1 silently re-runs the SAME request with max_tokens escalated (e.g. to 64K; turn count unchanged; emit a stream_restart event so UIs clear partial text). If still truncated, phase 2 commits the truncated assistant message and injects a recovery prompt whose wording is load-bearing: “Output token limit hit. Resume directly — no apology, no recap of what you were doing. Pick up mid-thought if that is where the cut happened. Break remaining work into smaller pieces.” — bounded by a recovery count, then accept the partial output. Related self-healing: on provider max-completion-token errors, regex-extract the provider’s actual limit from the error text, lower the effective cap, decrement the turn count, retry.

10. Structured output orthogonal to termination

“Are we done?” and “what did the agent produce?” are separate concepts: extract a typed payload from an XML tag in stdout, validated against a schema. Rules: caller owns the prompt-side instruction (engine throws early if the resolved prompt lacks the opening tag — misconfiguration guard); last match wins (agent self-correction is benign and frequent); fence-aware (unwrap ```json) but otherwise strict JSON — tolerant parsing hides genuine model failures. Validation failure throws a typed error carrying tag, raw match, cause, commits, branch, preserved workspace, and session id — side effects preserved, caller decides recovery. Retry = resume the same session with token-efficient error feedback so the agent re-emits a corrected tag without redoing the work (“resuming the session IS recovery”). The produce-then-extract two-phase pattern: run the expensive work without output constraints, then a cheap 1-iteration run resumes that session with a dedicated extraction prompt + schema + retries — keeps schema-emission pressure out of the work prompt and makes extraction independently retryable.

11. Model-call enforcement pipeline (output contracts + targeted retry)

Wrap every LLM call in an 8-stage pipeline: input guard → circuit breaker → token budget → prompt builder → caller → response validator → retry engine → fallback router, with per-attempt audit. Key pieces:

• Failure-mode taxonomy drives retry: enum {none, schema_violation, constraint_violation, prompt_injection, token_overflow, hallucination, timeout, circuit_open}; each mode maps to a “mutation hint” injected into the NEXT attempt’s prompt as a correction note (schema_violation → “Return ONLY a valid JSON object… start with ’{’”; token_overflow → “Aim for half the length”). Re-sending an identical prompt usually fails identically; blind retries waste money.
• No-retry modes as security policy: never retry prompt_injection (retries let an attacker brute-force pattern evasion) or circuit_open (hammers a dead backend).
• Declarative output contract: {must_be_json, required_keys, max/min_length, forbidden_phrases, must_contain}; auto-strip markdown fencing before JSON parse (fixes most format failures with zero retry calls); run cheap mechanical validation (~0.3ms) BEFORE any LLM judge.
• Three-state circuit breaker per backend: CLOSED → OPEN after 5 consecutive failures; OPEN → HALF_OPEN after 30s; any success → CLOSED; the is_open() check runs before any budget/prompt work so outage requests reject in ~0.05ms instead of hanging 30s (50 concurrent users × 30s = 25 min of blocked threads per minute of downtime).
• Slot-based token budget: reserve named slots in strict priority order (system_prompt → constraints → correction_hint → context → user_input) with real-tokenizer counts (char/4 heuristics are off by 40%+ on code/non-Latin text); context is the sole sacrificial slot, truncated to fit — instructions and correction hints are never crowded out.
• Audit every attempt: one JSONL record per attempt (correlation id, prompt hash, attempt number, failure_mode, latency, tokens, passed, strategy first|mutated_retry|fallback) powering pass-rate/failure-distribution/latency-percentile analytics.
• Foreground/background retry asymmetry: retry capacity errors (529) only when a human is waiting; fail background work fast to avoid load amplification. One flag threaded through the model-call layer.
• Hard timeout on every call (daemon thread + join; still-alive → typed timeout error, thread abandoned).

12. Permission models (four rungs, plus hooks as enforcement)

• Static rules: allow/deny lists with path/command normalization before matching.
• Hooks as deterministic lifecycle enforcement: shell scripts at pre-tool / post-tool / prompt-submit / session-start / stop / subagent-stop, receiving JSON on stdin, returning permissionDecision (allow/deny/ask), additionalContext (prepended for pre, appended for post), continue:false hard-stop; exit code 2 = block with error text the agent must fix. Aggregation precedence deny > ask > allow. Pre-tool hooks run BEFORE the permission check and can override it. Stop hooks can force continuation (“not done yet — keep going” injected as a user message) with a fired-once re-entry guard so a misbehaving hook can’t cause an infinite cycle. Uses: typecheck/lint/test after edits, block destructive bash (rm -rf, force push, DROP TABLE), auto-format, coverage-drop prompts.
• LLM permission classifier (auto mode): one cheap non-streaming call per proposed side-effecting action returns a binary verdict via a forced structured tool call (schema: thinking/shouldBlock/reason). Three deliberate hardenings: (a) the classifier’s transcript contains ONLY user messages + prior tool calls — assistant narration is excluded so the agent cannot influence the verdict through its own text output (anti-prompt-injection); (b) fail-closed: any API error / malformed verdict → block, never auto-allow; (c) a 3-tier category rubric — BLOCK-always regardless of stated intent (curl|bash, irreversible deletion, persistence mechanisms like .bashrc/cron edits, security weakening, sudo, network listeners), BLOCK-unless-clear-user-intent (writes outside cwd, system package mgmt, git push, >~20-file bulk ops), ALLOW (reads, local-only git, project-scoped tests/builds) — plus an aggregate-risk heuristic for rapid sequences of individually-safe actions and the rule “‘make it work’ does NOT authorize curl|bash”. Measured at ~84% prompt-reduction vs always-ask in one production deployment.
• Reviewer-agent adjudication: a middle rung where a separate agent evaluates each approval prompt instead of blanket bypass. Supporting patterns: headless/background sessions auto-deny “ask” verdicts but return a rich denial (why, allowed alternatives, “do NOT bypass; STOP and report the blocked action in your final summary for pre-approval”); “always allow” grants scoped to (operation type, target, session) and cleared on session reset; schedule-driven runs explicitly inherit an auto-approve policy rather than hanging on gates nobody will answer; plan approval pre-approves the permissions the plan implies; cross-process permission routing — a headless child’s permission prompt travels as a typed permission_request/permission_response message (with suggestions and modify-and-approve support) to wherever the human is. Approval UX spec: show what/why/what-could-go-wrong/if-approved/if-denied/modify-available; verbs approve/deny/modify/defer/always-allow-narrow/always-deny-narrow; log decisions as learning signals.

13. Instruction architecture (anti-bloat, router + metadata)

Giant instruction files fail via context-budget burn (600 lines ≈ 10–20K tokens), lost-in-the-middle (mid-file constraints get ignored — moving one security rule from line 300 to the top raised compliance 60% → 95%), priority conflicts (hard constraints look identical to style hints), and contradiction accumulation. Fix: the entry file is a router (50–200 lines: overview, run commands, ≤15 hard constraints, links with applicability conditions) + topic docs (50–150 lines each) read on demand. Every rule carries metadata: source (the failure that created it), applicability condition, expiry condition — audited and deleted like tech debt. Keep injected memory files SHORT and human-curated: a study of 138 agent-instruction files found LLM-generated files hurt performance while costing 20%+ more reasoning tokens; human-written ones helped only ~4%; directory overviews were useless. Structural placement beats prose: a numbered “Constraints (hard requirements, not suggestions):” block placed above the user content measurably reduced retry rates vs format rules buried in the system prompt. Split system prompts into explicit static/dynamic sections (identity/instructions vs per-turn cwd/date/git-status) for cache-friendliness. Long autonomous prompts should open with a reader contract (priority read order) and instruct the worker to write a compact operating summary to disk and re-read it at stage boundaries (self-recitation against context rot).

14. Tool design doctrine

Ten focused tools beat fifty overlapping ones — too many descriptions inflate the system prompt into “the dumb zone.” Prefer bash + one usage example in the instructions over a bespoke MCP server (one example command generalizes; a custom 6-example CLI replaced an MCP server and saved thousands of tokens). Freeze tool surfaces deliberately — a proven pattern keeps 9 tools with schemas/descriptions copied verbatim across platforms to prevent prompt drift. Budget observations with per-call overrides (e.g. an accessibility tree capped at 1200 nodes / 64 depth / 500 chars, expandable via explicit params, "max" opt-in) — default-compact, explicit-expand. Stateful external surfaces need a snapshot-freshness contract: snapshot once per turn before element-targeted actions; never reuse element indexes across turns; enforce mechanically (epoch-stamped snapshots) rather than by prompt discipline. Tool errors follow the WHAT / WHY / FIX envelope (“ERROR: fs import in renderer:12 / WHY: no Node API in renderer / FIX: move to preload, call window.api.readFile()”) — converts every failure into a self-correction opportunity without a human. Tool results cap size (default ~100K chars; truncate only text blocks, never slice base64 images). Machine-readable tool annotations (readOnly/destructive/openWorld hints) feed permission gating for unattended runs. Sensor output “optimised for LLM consumption” — linter messages with fix instructions are a positive kind of prompt injection. Security note: tool/skill descriptions are trusted prompt text and therefore injection surface — treat installs like random packages and fence descriptions, not just payloads (see security-and-guardrails).

15. Failure taxonomies (three complementary layers)

• Attribution (harness layer): every failure maps to one of five subsystems — task specification, context provision, execution environment, verification feedback, state management. Attribution decides the fix target (spec vs context vs env vs check vs state), and verification feedback is the lowest-cost/highest-return layer. A 4-layer runtime variant for diagnosing archived runs: routing (wrong task type) / execution (tool errors, permission blocks) / verification (output exists but doesn’t match) / governance (halted by safety check), applied by diffing against the last successful run (“evidence before intuition; compare, don’t guess”).
• Terminal-state taxonomy (7 states): baseline / keep / discard / crash / timeout / no-change / scope-violation — crash ≠ timeout (disambiguated via metadata), and no-change candidates inherit the parent’s scores without re-evaluation (zero wasted eval cost). Separately, three-state check outcomes: passed(score) / failed(score) / verifier-absent (None) — None is not 0.0: it can’t be promoted into a regression suite, reports as TIMEOUT not FAIL, and pass-rate denominators use the requested count so silently-dropped tasks count as failures.
• Runtime classification for supervision: classify child-agent fatal events into ProcessExited / SessionNotFound / Unknown vs rate-limited (regex 429|rate.?limit|quota → retryable status, no kill, no crash report). On real crashes, write a structured “testament” to the supervisor’s inbox (reason enum + last message) and wake it; a 60s inactivity watchdog treats hung workers the same; a short-window dedup table prevents double-finalize. Run-artifact contract making all this diagnosable: every archived run leaves manifest.json + execution_trace.jsonl (one tool event per line) + verification_report.json + failure_signature.json (failing stage + reason, machine-written at fail time).

16. Harness as optimization target (outer-loop search over harness files)

Treat the harness itself as the parameter: an outer loop where a proposer agent mutates a candidate copy of the harness files (instruction files, bootstrap/validate scripts) and deterministic validation + weighted evaluation decide keep/discard. Load-bearing mechanisms:

• Declarative target spec: one JSON file makes any directory optimizable — objective, constraints, baseline dir, tasks file + held-out test tasks file, required_files, allowed_write_paths, search mode (hill-climb | frontier), batch size, budget (outer iterations).
• Weighted deterministic task checks: two check types score instruction quality AND behavior in one rubric — file_phrase ({path, weight, required_phrases[]}) and command ({weight, command, expect_exit_code}); objective = hit_weight/total_weight.
• Change manifest → attribution (predict-then-verify): every proposal must declare {changes: [{component, files, failure_pattern, root_cause, targeted_fix, predicted_fixes[], risk_tasks[]}]}; after evaluation, diff parent-vs-candidate task results into fixed[]/regressed[] and score each declared change: verdict EFFECTIVE / PARTIALLY_EFFECTIVE / INEFFECTIVE / MIXED / HARMFUL, plus unattributed_regressions (the scariest class). Missing manifests are lenient-but-recorded (flagged, not hard-failed).
• Experience directory (filesystem-as-memory): copy ALL prior candidates’ artifacts, scores, and traces into the current candidate’s visible workspace. The core research finding: an agentic proposer reading raw prior source/scores/traces beat compressed-text optimizers (+7.7 pts online classification at 4× fewer context tokens; +4.7 pts on held-out math RAG across 5 models). Give refiners raw artifacts, not summaries.
• Dual acceptance gate (from the sibling ratchet system): (1) regression suite pass-rate ≥ threshold (0.8) on a self-growing suite — starts EMPTY, grows only when a previously-failing task passes AND re-verifies (grew 0 → 17 cases over 18 batches; late iterations rejected most candidates, making improvements “genuinely additive”); (2) monotonic held-out score ≥ best-ever recorded. Discipline rules: one hypothesis per iteration; revert on gate failure; abandon a hypothesis after 3 failures; halt after 5 consecutive non-improving iterations, write a summary, surface findings.
• Write-scope enforcement by diff, three layers: advisory (allowlist in the prompt) → authoritative (post-hoc workspace diff; any out-of-scope change ⇒ scope-violation, candidate dead regardless of score) → optional OS pushdown (sandbox write_paths). Same conclusion reached independently by a git-based file guard (git diff-index vs an allowlist frozenset, re-checked on HEAD~1..HEAD so commit-first can’t bypass it): enforce editable surface structurally, never by instruction.
• Structural information hygiene: test/held-out traces are never written to disk in reach of the optimizer (env-flag disabled at the writer) — store what must not be read where it structurally cannot be read; held-out evaluation runs only post-loop on the frontier.
• Pareto secondary objective = context cost: among equal scorers, prefer the smaller context footprint.
• Fake deterministic backend as pipeline CI: a no-model proposer exercises the whole loop so the orchestration matures before burning model time. See self-improvement-loops for the flywheel side (failure mining, clustering, lesson promotion).

17. Subagents as context firewalls (not roles)

Role-based subagents (“frontend engineer”) don’t work; context control does. The parent sees only the prompt and a condensed result, never intermediate tool calls; subagents cite sources in filepath:line format so the parent can drill down without re-including transcripts. Cost lever: expensive model for the orchestrating parent, cheap models for subtasks. Continue-vs-spawn is a policy, not a vibe: continue an existing worker when its explored files overlap the edit target or it is correcting its own failure (it has full error context); spawn fresh for narrow implementation after broad research, independent verification, or wrong-approach retries. High overlap → continue; low overlap → spawn. Re-delegation prompts must contain file paths, line numbers, exact changes — “based on your findings” is banned as lazy delegation. Deeper orchestration (mailboxes, task boards, testaments) belongs to multi-agent-orchestration.

18. Deterministic pre-work: bootstrap snapshots and dry-run preflight

Before any agent session: a deterministic, LLM-free environment bootstrap snapshot (top-level dir entries capped ~20, which package files exist, which()-probed tools, git branch+status capped 20 lines, platform/memory) written as summary.md + snapshot.json and embedded in the prompt — measurably reduces redundant exploration turns. Before any run: a dry-run readiness verdict that statically resolves settings, auth, prompt assembly, skills, tools, and config — no model calls — and reports ready/warning/blocked with next_actions. Probe-before-run (auth/latency smoke) before committing a full run.

19. Harness simplification and ablation

Every component compensates for a past model weakness; assumptions expire. Practice: monthly, disable ONE component → benchmark → delete/restore/lighten. Radical one-shot removal fails and obscures which pieces were load-bearing; methodical one-at-a-time removal works. Real sequence: sprint-splitting removed when a newer model decomposed natively (the builder then ran 2h+ without drift) but the evaluator kept (still caught stubs near the capability boundary); evaluator value is task-dependent — “worth the cost when the task sits beyond what the current model does reliably solo.” Tag every harness behavior with the failure mode it mitigates and the model era it was added in, so upgrades can retire scaffolding methodically.

20. Sandbox and workspace lifecycle

Two provider kinds behind one tiny handle contract: bind-mount (host creates a git worktree, provider mounts it — no sync) and isolated (own filesystem, code synced via git format-patch + git am --3way, with a sandbox-owned ref tracking the last-synced commit since am re-commits new SHAs); noSandbox() as a first-class escape hatch. Handle = exec(cmd) → {stdout, stderr, exitCode} (non-zero returned, not thrown), close(), worktreePath, copy-in/out — custom providers in ~50 lines. Worktree discipline: reuse-by-default with ff-only-when-safe refresh (never reset --hard — unpushed agent commits must never be clobbered); PID-liveness file locks stored outside the worktree (atomic O_EXCL; live PID → fail fast, dead → self-heal); dirty-preservation on close (a worktree with uncommitted changes survives, path surfaced; fail-closed to dirty on error); deterministic branch naming so retries are idempotent. Container permissions pitfalls: align UID via image build-arg (never runtime recursive chown — it walks into bind mounts), pre-create parent dirs of single-file mounts, SELinux :z labels, --userns=keep-id for rootless. Fork must declare which axes it isolates (session-only fork leaves branch/worktree shared — concurrent forks race unless each gets a distinct branch); unique-name generators need randomness, not just timestamps.

21. Prompt assembly as a fail-fast security boundary

Dynamic context commands embedded in prompt templates execute inside the sandbox, after setup hooks. Expansion never retries or degrades: a timed-out or failing expansion aborts the run, because a prompt assembled in a degraded environment burns an iteration and possibly commits garbage — worse than a clean abort in AFK contexts. Retry is safe for idempotent infrastructure, unsafe for prompt content; typed diagnostics (elapsedMs, exitCode) go to the orchestrator, which owns the retry decision. Injection rules: interpolation/expansion only for template-sourced prompts (programmatically-built strings are literal — embedded issue bodies may coincidentally contain template syntax); expansion markers inside substituted values are inert text, so user-authored content passes safely through args.

Patterns & compositions

• Planner / generator / evaluator split. Planner expands a 1–4 sentence prompt into a full spec and stays at product altitude (granular technical detail in plans cascades errors downstream; without a planner, generators under-scope). Generator works in sprints, one feature at a time. Evaluator acts (drives the live app, runs tests) rather than reads, with hard per-criterion thresholds. Communication is file-based. 5–15 iterations per run; details of evaluator calibration live in verification-and-judging.
• Sprint contracts. Before each work unit, generator and evaluator (or human) negotiate what “done” means — scope / verification standards / exclusions — iterating until agreed; the gate later enforces exactly that contract. “Writing the done condition first caught more scope drift than any prompt change.”
• Harness as state machine. Explicit phases with entry/exit criteria, structured schemas validated at every phase boundary, validation loops (not just final summaries), artifacts recorded per stage, resumable mid-run; checkpoint at step level (“never force a workflow to restart because phase seven failed”); idempotent effect layer for side effects (idempotency keys, attempted/committed/compensated/skipped records) — the workflow-engine generalization lives in workflow-engine-design.
• Deterministic-gate interleaving. LLM step → hard-coded gate the agent cannot skip → LLM fix step → hard-coded commit. Reliability comes from the quality of the constraints, not the size of the model.
• Cheap-validation-before-expensive-evaluation. A deterministic existence/non-emptiness/schema validator ahead of every LLM judge; invalid candidates never reach evaluation. Gate commands run shell=False via shlex, metacharacters rejected unless the spec opts in; missing-prerequisite commands are skipped, not failed.
• Failure → artifact method map. Add only the smallest artifact that targets the observed failure: cold-start confusion → progress file; scope sprawl → feature list; premature completion → clean-state checklist; fragile startup → init script; weak handoff → handoff template; subjective review → evaluator rubric.
• Review-feedback promotion. Every recurring review comment becomes an automated check; “capture human taste once, enforce continuously — when docs aren’t enough, promote the rule into code.” Background cleanup workflows open small auto-mergeable PRs against golden rules encoded in the repo.
• Layered observability. Runtime observability (logs/traces — what did it do) + process observability (plans, contracts, rubrics — why should this be accepted). Missing observability costs 30–50% of session time on redundant diagnosis. Trace trajectories, not only outcomes — a system that gets the right answer through a dangerous path is not yet reliable. Event forwarder errors are swallowed: a broken subscriber must never kill the run.
• BYO-runner seam. A 4-method proposer/runner contract (name / prepare / invoke / collect) with a normalized result (applied, changed_files, events[], token_usage, cost, files read/written, tool_call_count) + per-backend instruction rendering (one canonical instruction object rendered into each runner’s native file dialect); plus cross-runner conformance smoke — one deterministic fixture, one scenario set, run through N agent runtimes. Catalog data (per-runner capability rows, health-check evidence columns, pinned checksummed adapters) belongs to ecosystem-and-interop.
• Harness templates per work topology. Bundle guide + sensor pairs (feedforward doc + matching gate/judge) per topology (CRUD service, event processor, dashboard) — a variety-reduction move; codebases differ in “harnessability” (typed languages are a free sensor), and the harness is most needed where it is hardest to build.

Anti-patterns & failure modes

• Self-grading / premature victory declaration. Agents reliably skew positive on their own work; an untuned judge “identifies real issues, then talks itself into approving anyway.” Detector: a check that has never once rejected across hundreds of turns is statistical proof it isn’t a real check. Fixes in verification-and-judging.
• Context anxiety. Some model tiers rush and skip verification as the window fills; compaction does not cure it — only resets do. Harness mitigations are model-specific and become dead code on upgrade (an entire mitigation layer for one model generation was obsoleted by the next).
• Instruction bloat / LLM-generated memory files. Mid-file constraints get ignored (lost-in-the-middle is measurable: 60% → 95% compliance by position alone); LLM-generated instruction files hurt performance at +20% token cost.
• Stateless interval loops for cumulative work. Rerunning “keep implementing X” every 10 minutes fails because iterations share no memory — a designed-in failure. Cumulative loops must thread external state (disk) between iterations.
• Verbose success output. Thousands of lines of passing-test output flood context and cause hallucination. Success is silent; only errors are verbose.
• Radical one-shot harness simplification. Removing many components at once fails and hides which were load-bearing; ablate one at a time.
• Role-based subagents. Persona framing delivers nothing; context isolation is the actual mechanism.
• Read-and-mark message loss. Atomically marking queue messages read before the consuming turn completes loses them if the agent crashes mid-turn; use peek → deliver → mark-read-on-success.
• Silent retry storms / blind retries. Identical-prompt retries fail identically; retry storms destroy operator trust. Retries are failure-mode-classified with mutation hints; repeatedly-failing work goes to a dead-letter queue with evidence-rich explicit replay; retry once then change strategy or escalate.
• Retrying blocked injections. Auto-retrying a payload that tripped the injection screen lets an automation loop brute-force its own guard.
• Degraded prompt assembly in AFK runs. Best-effort expansion markers and retry-on-expansion-failure feed the agent a prompt built in a broken environment — abort instead.
• Mixing initialization with implementation. A broken multi-objective problem: agents favor visible feature code over infrastructure; separate the phases.
• Overreach/under-finish. k simultaneous tasks each get ~C/k context; below a threshold nothing finishes; more LOC correlates with fewer completed features. WIP=1.
• Trusting the agent’s self-summary. A run’s self-report “is not the source of truth” until service-level verification completes; no task is complete solely because the agent says so.
• Conservative-flag inversion. Safety-guard config flags must parse missing/null/unknown values as ON; only explicit falsy disables — a typo must fail safe.

Quantitative findings

• Same model, bare vs full harness: 20 min/$9/broken vs 6 h/$200/working; simplified harness on the next model generation: ~3 h 50 min/$124.70, QA rounds ~$3–4 each.
• Harness-only optimization with frozen model: benchmark 0.560 → 0.780 (~40% relative) over 96 autonomous experiments; regression suite self-grew 0 → 17 cases; proven stop values: abandon a hypothesis after 3 failures, halt after 5 non-improving iterations.
• Engine-owned completion verification lifted real completion 37.5% → 87.5% in one measured case.
• Team success on fixed model via harness layers alone: 20% → 60% → 80% → ~100% (instructions → verification commands → progress files).
• WIP=1 (“small next step”): +37% completion; 800 lines/20% pass vs 200 lines/100% pass (LOC negatively correlates with completion).
• Dedicated initialization session: +31% feature completion, recouped in 3–4 sessions.
• Handoff/continuity discipline: rebuild cost −78% (target ≤3 min to executable state), completion 58% → 100%, hidden defects 43% → 8%.
• Clean-exit strategy over 12 weeks: builds passing 68% → 97%, tests 61% → 95%, session startup 60+ min → 9 min.
• Constraint position: moving one rule from line ~300 to the top of the instruction file raised compliance 60% → 95%.
• 138-agentfile study: LLM-generated instruction files hurt performance at 20%+ more tokens; human-written helped only ~4%; directory overviews useless.
• Sprint contract + rubric evaluation: 3–4 blind retry cycles (~45 min) → 1 iteration (~15 min); evaluator prompts need 3–5 tuning rounds against human judgment.
• Harness overfitting: same model ~#33 in its native harness vs ~#5 in a custom harness on the same benchmark; a team went Top-30 → Top-5 changing only the harness.
• Agentic proposer over raw prior artifacts vs compressed-text optimizer: +7.7 pts (online classification) at 4× fewer context tokens; +4.7 pts on held-out math RAG across 5 models.
• LLM permission classifier: ~84% reduction in permission prompts in production.
• Circuit-breaker outage math: 50 concurrent users × 30s timeout = 25 min of blocked threads per minute of provider downtime; open-circuit rejection ~0.05ms.
• Enforcement-pipeline benchmark: 55% first-attempt failure rate → naive 0% vs pipeline 100% eventual pass (median latency 43 → 144ms; ~2.9ms non-LLM overhead); 2/10 successes were fallbacks — the layer doesn’t fix a bad model.
• char/4 token heuristics are off by 40%+ on code/non-Latin text; count with a real tokenizer.
• Missing observability costs 30–50% of session time on redundant diagnosis; machine-readable progress records cut startup diagnosis 60–80%.
• Production deterministic-rails pipeline: >1,300 machine-written PRs merged/week, 1,000+ concurrent disposable sandboxes, every PR still human-reviewed.
• Loop-engineering defaults that recur across systems: idle timeout 600s; completion grace 60s; max ~3 fix attempts; stagnation trip at 3 identical errors; escalate after 5 consecutive failures; tool-result batch cap 10; compaction breaker at 3 failures; task hard timeout ~30 min; sub-delegation depth cap ~5.

Open questions

• Harness coverage metrics. No one has an analogue of code coverage for harnesses: which failure classes have a sensor (test, gate, judge, drift trigger) and which are unwatched, silent-sensor = quality or blind spot?
• Trace-driven self-repair. Agents analyzing their own run traces to propose harness-level fixes (new gate, hint, trigger) — named as an open problem; the change-manifest/attribution loop is the closest working piece.
• Just-in-time harness assembly. Harness as “something closer to a compiler” that assembles tools + context per task rather than a static configuration.
• Coherence at scale. Keeping a growing harness (accumulated rules, hooks, gates) internally consistent; rule metadata (source/applicability/expiry) helps but no system demonstrates automated contradiction detection.
• Parallel agents on shared codebases. Resource-claim protocols and worktree isolation exist, but merge-conflict-aware scheduling across many concurrent agents is unsolved (see multi-agent-orchestration).
• Where does the enforcement pipeline end? The model-call chokepoint (contracts, breakers, budgets, audit) is proven for single calls; whether streaming outputs and multi-turn tool loops can be contract-validated mid-flight (streaming validation) is undemonstrated.
• Budget semantics. Every system agrees caps must exist before unattended runs (“a loop without caps has delegated its spending authority to its own bugs”), but no corpus source formally defines budget composition across nested runs (parent run vs subagents vs retries).