Self-Improvement Loops

Part of the PersonalClaw research-learnings library. Source-agnostic; distilled 2026-07-13 from a 95-source competitive-research corpus.

Principles (the durable truths — one paragraph each)

The ratchet doctrine. Any time the agent makes a mistake, engineer a solution such that it never makes that mistake again — and add constraints ONLY after a real observed failure (“ratchet, don’t brainstorm”). Every injected rule must be traceable to a specific thing that went wrong, carry provenance to the failure that created it, and have a retirement path when a more capable model makes it redundant (validate removals by ablation). Corollary: the right harness cannot be downloaded — it is shaped by your failure history. Multiple independent systems converge on this as the trust anchor of any learning flywheel.

Traces are the universal substrate. Every learning mechanism — routing policies, skill optimization, config evolution, spec search, skill discovery — should be a consumer of one append-only trace/run-ledger store that records per-step identity, tool calls with outcomes, tokens, latency, cost, and which skills/artifacts were actually consulted. Multiple independent systems converge on this architecture; systems without it have nothing to learn from.

Self-report is structurally untrustworthy; acceptance must be external and statistical. Builder agents systematically overestimate completeness (“a model is its own output’s best defense attorney”); single-run LLM-judge acceptance is statistically indistinguishable from noise. Working systems externalize completion to the engine (worker may only request a transition; the engine executes verification and flips state irreversibly — one measured case lifted real completion 37.5%→87.5%), and gate self-modification with statistics: median-of-N judging with an epsilon margin, held-out regression checks, and monotonic best-ever ratchets.

Harness-level fixes are the cheap lever. With a completely frozen model, pure harness edits (prompts, state tracking, context construction, tool interfaces, workflow design) produced a 0.560→0.780 validation score (~40% relative) over 96 fully autonomous experiments — the strongest open quantified evidence that the optimization target should be the harness, not the weights. A separate team case went 20%→60%→80%→~100% success purely by adding instructions → executable verification commands → progress files.

Propose, don’t write — and make application revertible. Optimization output should land as proposals (a queue, a proposals directory, a sidecar overlay) rather than in-place mutations of live artifacts. Revert must be trivial (delete a file, roll back a version), corrupt output must be unable to break the base artifact (fault-tolerant overlay loading), and every accept/auto-enable decision must be audited. Multiple independent systems converge on this.

Attribute failure before repairing. A failure is caused by the SKILL (wrong/missing guidance → edit it), the AGENT (misuse, didn’t read available correct guidance → do NOT bloat the skill), or the ENVIRONMENT (flaky API → at most a brief instability note). A richer 5-layer decomposition (task specification / context provision / execution environment / verification feedback / state management) and a 13-class gap taxonomy (missing skill / tool / permission / memory, bad decomposition / verification, unsafe autonomy, poor routing, context overload, weak observability, missing eval, external dependency, bad requirements) turn “it failed” into a targeted, most-leverageful repair. Named anti-rule: if correct info existed and the agent ignored it, never delete the correct facts in favor of “go discover it”.

Risk-tier every self-edit deterministically. Edit type — not judgment — assigns the tier: parameters/routing/tools → auto-apply if gates pass; prompts/agent-class/few-shot → queue for human review; model weights / high-risk capabilities → never auto-applied. Auto-promotion is permitted only for low-risk candidates after a static preflight, and every auto-enable decision is logged for audit.

Every success should leave a ratchet behind. “If a task succeeds but leaves no reusable artifact (skill, template, eval, monitor, policy, memory), some of the value is being lost.” A completing run emits ≥1 improvement candidate or an explicit “no ratchet” with reason; a system-owned improve queue (eval gaps, flaky templates, repeated failures, stale assumptions) is drained during idle time.

Mechanisms (implementation-ready designs)

1. Two-layer run evidence for the optimizer

Attach to every session/run: (a) _trajectory — a programmatic, lossless, zero-LLM step trace: [Step N] | score=x | read_skills=[..], each tool call as name(args) → ✓/✗ outcome, every field clipped to ~400 chars, max 8 tools per step, user prompt deduplicated (shown once); (b) _summary — one cached LLM “trajectory-aware analytical summary” (8-15 sentences) explicitly prompted to preserve SEQUENCE and CAUSALITY: goal, key trajectory (“read skill X → tried Y → hit error Z → switched to W”), per-skill effectiveness (helped/hurt/missing/wrong), turning points, outcome. Both are attached once and reused by every downstream stage; evolution prompts mark the trace as ground truth over the summary. Compresses MB-scale raw sessions to tens of KB. Cap ~30 sessions per evolution prompt. See verification-and-judging for the judge side of consuming these.

2. Evaluator-optimizer core loop (teacher/student spec search)

A frontier “teacher” model acts as meta-engineer for the local “student’s” harness (prompts, routing, tools — never weights). Loop: Diagnose (teacher reads traces read-only, identifies 2-5 failure clusters tagged (student_failure_rate, teacher_success_rate, skill_gap)) → Plan (typed edit list with deterministic risk-tier assignment) → Execute (per edit: apply → benchmark on a held-out subsample, default 50 → git-backed checkpoint commit or rollback) → Record (learning session to durable store + JSON artifact). Acceptance predicate GateOK: the target cluster must improve AND every other cluster may regress at most eps (default 1%). Whole-cycle acceptance: post-eval − baseline ≥ min_improvement (0.02), else rejected. Sessions repeat until gate-score stagnation (k=5 consecutive, eps=0.001) or cost budget exhaustion.

3. Statistical acceptance machinery (the anti-noise kit)

For a skill/prompt-body optimizer where the artifact is the trainable parameter: forward pass on a training batch → two reflect calls (over failures AND successes) propose edits → rank and clip to a learning-rate budget (default 4 edits/step, cosine schedule) → tagged-anchor patching (ambiguous anchors → rejected buffer) → validation gate: median-of-3 judge runs + epsilon; accept only if median beats best by >0.05 → crash-safe atomic commit every 5 steps. Guards, all load-bearing: held-out gate (disjoint task IDs; a candidate that beats the benchmark but regresses held-out is refused), frozen region (frontmatter/trigger metadata never mutated — prevents routing drift), read-only tool sandbox during rollouts, cost preflight (default $5 cap), per-artifact DB lock, dirty-tree refusal, and a bootstrap review sentinel: generated benchmarks carry a # BOOTSTRAP_PENDING_REVIEW marker a human must delete, preventing self-referential benchmark gaming. Artifacts: best.md, versioned snapshots, append-only history.json and rejected.json. Measured cost: ~$0.71-0.90 per 20-task optimization run. The core finding motivating median-of-3: single-run LLM-judge acceptance was statistically indistinguishable from noise.

4. Dual gate + self-growing regression suite

Four-step deterministic gate (exit 0/1 = accept/revert) for autonomous harness iteration:

• Step 0 — file guard: git diff vs HEAD + untracked files; anything outside a frozen ALLOWED_FILES allowlist fails. Re-checked over HEAD~1..HEAD at record time so committing forbidden files first cannot bypass it. Conservative-by-design flag parsing: only explicit falsy disables the guard.
• Step 1 — regression suite: re-run suite.json tasks (pass = reward ≥ 0.5); pass rate ≥ threshold (0.8); denominator is the requested task count so silently-dropped tasks count as failures. Empty suite skips (bootstrap).
• Step 2 — monotonic validation (always runs): full held-out run; score must be ≥ best-ever recorded. The no-regression ratchet.
• Step 3 — suite promotion (only after 1+2 pass): previously-failing tasks that now pass are re-run; if they pass again they merge into the suite. A None (verifier-never-ran) result can never be promoted. The suite starts EMPTY and grows only with genuinely conquered tasks — measured growth 0→17 cases over 18 batches; late iterations rejected most candidates, making surviving improvements “genuinely additive.” This is eval-as-living-distribution at personal scale, answering the objection that golden-run suites are team-scale CI machinery. Companion semantics: reward None ≠ 0.0 (infra error/timeout, reported TIMEOUT not FAIL); immutable baseline/ traces (write-once) beside latest/ (overwritten) so every iteration can diff against first-run behavior; 70/30 train/test split stratified by pass/fail so both splits carry comparable difficulty; test traces are never written to disk (structural hygiene — see anti-patterns).

5. Predict-then-verify change attribution

Every self-modification proposal must declare a change manifest: {candidate_id, parent_ids, changes: [{id, component (required), description, files[], failure_pattern, evidence_refs[], root_cause, targeted_fix, predicted_fixes[] (check ids this should fix), risk_tasks[] (check ids it might break)}]}. Validation is lenient-but-recording (missing manifest → warning + flag, not hard fail). After evaluation, diff parent-vs-candidate per-check results → fixed[]/regressed[], score each declared change (actually_fixed = predicted ∩ fixed, risk_realized = risks ∩ regressed, hit_rate) and assign a 5-way verdict: HARMFUL (risk realized, nothing fixed) / MIXED / EFFECTIVE (all predictions fixed) / PARTIALLY_EFFECTIVE / INEFFECTIVE — plus unattributed_regressions (regressions nobody predicted: the scariest class). Verdict history per proposal source becomes a trust signal; HARMFUL verdicts auto-generate revert proposals. Pair with a 7-state candidate outcome taxonomy: baseline / keep / discard / crash / timeout / no-change / scope-violation — no-change candidates inherit the parent’s scores without re-evaluation (zero wasted eval cost), and scope violations are computed by authoritative post-hoc workspace diff against an allowed_write_paths list (prompt advisory + diff check + optional OS-sandbox pushdown), never by instruction alone.

6. Experience directory: filesystem-as-memory for the proposer

Before each proposal, copy into the candidate workspace ALL prior candidates’ artifacts (source, scores, traces), a parent summary, and an index; tell the agent to inspect them. The headline research finding: an agentic proposer with filesystem access to full prior candidates beat compressed-text optimizers by +7.7 points at 4x fewer context tokens (online classification) and +4.7 points on held-out models (math RAG). Give refiners raw prior artifacts, not LLM-summarized digests. A related shipped mistake to avoid: one optimizer fed each round the stale baseline eval report instead of the latest round’s — feed the freshest report.

7. Sidecar overlays as revertible application

Optimization output is stored as a sidecar file (e.g. <artifact>/optimized.toml: metadata + few-shot exemplar pairs), never mutating the original. The loader is fault-tolerant — a bad overlay must not break base loading — and overlays override descriptions / inject exemplars at prompt-assembly time. Revert = delete the file; diffs are inspectable by construction. Complementary version-registry shape: name → {id = sha256(name)[:12], monotonic version, content sha, file records, history capped at 20 entries}, with every version’s full bundle archived.

8. In-bundle evidence ledger + read-before-write

Before ANY edit to a learned artifact, snapshot the current body to history/v<N>.md and write history/v<N>_evidence.md: decision summary, motivating run/session IDs with scores and failure patterns, retrospective on prior edits (did they help/hurt/inconclusive?), exact sections preserved vs changed, open questions for future rounds. Reading ALL history before deciding is mandatory, not optional — skipping it causes the named failure “reverting past improvements” (oscillating edits). Version-numbered filenames only (dates are too coarse when multiple rounds run per day). Rejected diffs are kept as negative training signal and audit trail. Lineage fields on entities (iterates edges, is_current_best with an at-most-one-per-lineage invariant, eval_score) make the empirically-scored chain explicit.

9. Four-action evolution decision space + conservative-editing doctrine

Per evidence group, ONE combined decision+execution call chooses among: improve_skill (targeted edits), optimize_description (rewrite ONLY the trigger/description — body untouched; wrong-triggering is its own repair class because the description IS the retrieval surface), create_skill (pattern distinct from existing; must differ from all existing names), skip (“when in doubt, prefer skip”). Conservative-editing constraints worth embedding near-verbatim in refiner prompts: current artifact is source of truth, not a draft; targeted edits over rewrites; preserve structure/headings/terminology; keep factually-correct details even if the agent misused them; hard bans on changing ports/endpoints/contracts without clear evidence, on whole-artifact rewrites, on imposing templates, on adding generic best practices (retries/caching) the model already knows. Skills = compressed environment information (endpoints, quirks, procedures), imperative, evidence-driven, “reusable guidance, not a failure postmortem”, with descriptions stating trigger contexts including explicit NOT for: ... exclusions.

10. Reject-by-default publication verifier

A post-generation, pre-publish LLM gate whose job is explicitly NOT to improve the artifact. Approve only if ALL named checks hold, each with its own score: grounded_in_evidence, preserves_existing_value, specificity_and_reusability, safe_to_publish. Overall score falls back to mean-of-checks; reject by default on LLM failure or unparseable JSON; acceptance threshold 0.75. Check scores are stored on both accepted and dropped proposals so every drop is auditable.

11. A/B replay validation against a baseline

Before publishing an edit, mine up to 3 replay cases from real turns (prefer tool-free turns); re-ask each instruction twice — once with the current artifact in the prompt (baseline), once with the candidate — score both with a process-reward judge (majority of M=3 votes at temp 0.6 in capture; temp 0.1 for replay; normalize −1..1 → 0..1, unclear → 0.5); accept iff candidate_mean >= threshold AND candidate_mean >= baseline_mean. At fleet scale, distribute as validation jobs picked up by idle opted-in clients with quorum publish thresholds (min_results, min_approvals, min_mean_score, reject at max_rejections); validation workers are disabled by default, idle-gated (idle_after_seconds), daily-quota’d, and concurrency-capped. At N=1 scale, a single replay-vs-baseline result attached as evidence to the proposal card still beats a pure judge opinion.

12. Baseline acceptance gate for generated workflows/templates

A generated template must beat just asking the model once: compare the candidate against a single-model one-pass baseline with a hard quality threshold (≥0.80) on a weighted rubric (40% body usefulness / 25% trigger+IO specificity / 20% gate+risk+collision coverage / 15% reusability). Full gate ladder for creator pipelines: collision check (generic/overlapping triggers flagged) → lint → risk classification (machine-assigned low/medium/high + capability list) → smoke on cheap models → runtime E2E (eval prompt derived from the candidate’s own trigger) → acceptance-compare → persist as a proposal, never a live artifact. A creation-mode ladder (PREVIEW_ONLY / PERSISTED_PROPOSAL / FULL_GATED) trades gate cost for confidence.

13. Three-arm eval with a placebo control

To validate that a skill/lesson/SOP’s specific content works: run scenarios under three arms — (a) the entity, (b) nothing, (c) a same-length generic placebo — with blind subagent grading and N≥3 trials, capturing verbatim rationalizations from failures. Beating “nothing” only proves more text helps; beating the placebo proves the content. The trivial-control arm is the load-bearing innovation. Related protocol (“vibe tests”): evaluate with fresh, context-free subagents; evaluation-only expected outcomes are never shown to the executing agent; only the artifact under test varies; attach 2-3 probe prompts per artifact (“if the agent can’t answer these, it isn’t loaded” — 0% pass without the doc is the calibration point).

14. Failure mining → clustering → prioritized proposals

Convert failed-run traces into structured failure records with root-cause hypotheses; cluster by shared root-cause mechanism; rank clusters by (total failures × low resolution rate); target the optimizer at the highest-priority unresolved cluster — not the most recent failure. One measured deployment auto-discovered 29+ distinct failure clusters without labeling. Machine-write a failure_signature record at failure time ({failing_node, stage, layer, reason, input_hash}) so mining is mechanical.

15. Measured effectiveness feeding retrieval (close the loop without an eval harness)

Track per artifact: inject_count, positive/negative/neutral outcome counts (from per-turn process-reward feedback on turns where it was injected), effectiveness = positive/injected (default 0.5 unknown). Blend into retrieval ranking: score = similarity * (0.3 + 0.7 * effectiveness), then prune near-duplicates (pairwise embedding sim > 0.9, keep the higher-weighted). Critical attribution rule — injected ≠ used: prompt-time injection alone is never evidence a run used an artifact; only actual reads/loads (derived by mapping tool-call file paths to artifacts) count for evolution grouping and credit.

16. Volunteered-vs-used measurement (the minimal viable flywheel metric)

Log every surfacing event with its resolution arm and per-arm confidence (proven bases: alias 0.9, exact title 0.8, fuzzy/slug 0.6, +0.05 recency/frequency bonus; gate at 0.7; cap 3 surfaced items); “used” = the artifact was actually retrieved/loaded after being volunteered. Report per-arm precision and tune thresholds from it (“use per-arm precision to tune min_confidence, not as an exact metric”). Store events as deterministic template strings (never raw conversation text), prune at 90 days. Cheap, requires no eval harness, and makes threshold calibration empirical.

17. Trace mining for skill/template discovery

Two complementary miners: (a) scan the trace store for recurring successful tool sequences above min_frequency and min_outcome quality thresholds → draft new skill manifests into a discovered/ staging dir; (b) idle-time harvest of N-day (e.g. 30-day) skill/action co-occurrence from run history to seed template proposals, running only in unattended windows. A third source: intent inversion — after each run, one cheap LLM pass over (goal + node names + final summary) synthesizes a canonical 120-200-word user-register intent; embed it; ≥k near-duplicate intents with no matching template → template-suggestion proposal carrying the synthesized intent as description/match text. Embedding the normalized synthesized intent (not raw transcripts) gives a register-normalized similarity space for dedup/clustering. See knowledge-pipelines.

18. Proposal queues and decision memory

The queue mechanics that make propose-don’t-write livable: content-signature dedup (normalized-content SHA; duplicates refresh the existing entity instead of filing new proposals), supersedes lineage, soft-delete (disabled-not-deleted preserves audit), sha-based conflict detection when two accepted proposals target one entity with LLM merge (“preserve ALL actionable guidance from both; on contradiction prefer the more specific; merged description covers both trigger sets”; merge failure → keep incoming). Decision memory for approvals: actions carry a tier (trivial/low/medium/high) and a permission_key; user replies like “always no {id}” persist as pattern-keyed always-approve/always-deny rules (e.g. email_delete:domain:noreply.github.com) so the same decision is never asked twice; anti-hallucination rule — referenced IDs must be copied exactly from the source digest; seen-ID dedup + stale expiry per run. Approval decisions themselves are learning signals. Approval UX spec: show what/why/what-could-go-wrong/if-approved/if-denied/modify-available, with verbs approve / deny / modify / defer / always-allow-narrow-scope / always-deny-narrow-scope.

19. Trajectory-variance detection (two-way tier migration)

Over the run ledger, compute per-template trajectory variance (node paths taken, tool-call sequences, iteration counts). Low-variance agentic templates are “workflows wearing a costume” → propose distillation into a fixed workflow (cheaper, more reliable). High-failure fixed steps → propose promotion to agentic stages. A two-way migration proposal class alongside content diffs. See workflow-engine-design.

20. Canary windows and drift-aware measurement

After applying an accepted diff, treat the next N runs as a canary window: compare outcome metrics against the prior version and auto-revert (or file a demotion proposal) on quality regression. Separately, run a fixed daily canary suite against model providers and alert on regression, keeping “model quality” separate from “serving reliability” — because measured day-to-day provider variance is ±8-14% (MoE/batching), a 5% real change is indistinguishable from noise by feel. Pin model snapshots; store per-request metadata (model id, params, latency, quality label). This drift bound is also why single-run acceptance judging fails (mechanism 3).

21. Triad proposal search + shape-assertion scoring

Generate prompt/template revisions in triads: conservative (surgical edits to current best), moderate (recognizable but new concepts), wild (full redesign, existing artifact disposable). Score against journaled histories with deterministic response-SHAPE assertions (must_match_any acceptable next moves, forbidden-text scoped to prose regions, syntactic-validity checks; never execute model-produced code / never re-fire side effects), keep a rolling leaderboard with human review notes. Acceptance is strictly-greater-than-best; rejected candidates are recorded but never built upon. Pareto selection with context-cost as the secondary objective — between equally-scoring candidates, the one with the smaller context footprint wins.

22. Self-improvement operating modes + abandonment ladder

Two modes: (1) inline after every task — record worked/failed/slow, classify the gap (13-class taxonomy), update the smallest useful artifact, add an eval if a blind spot was exposed; (2) background — ONE improvement hypothesis, one bounded change, eval slice vs baseline, keep-if-better/revert-if-worse, log. “Never do giant prompt surgery without eval protection.” The abandonment ladder (proven values): one hypothesis per focused change per iteration; on gate failure revert immediately; after 3 failures of the same hypothesis, abandon it; stop after 5 consecutive iterations without validation improvement, write a summary, surface top findings. Append a learnings log every iteration, pass or fail, including a structured “needs from human” field — issues the agent cannot resolve get surfaced, never silently retried.

23. Health-scored, budget-capped self-remediation

Instead of N independent maintenance crons: one doctor-style job computes a health score over measured deficits, builds a dependency-ordered remediation plan (sync before extract, embed after consolidate), re-checks the score per step, and refuses to spend past a cost cap (--target-score 90 --max-usd 5 semantics). max_reachable_score ceilings (e.g. missing embedding key caps score at 60) prevent futile spend. Autopilot cadence adapts: healthy score → sleep longer. Routing rule: deterministic work → job queue; judgment work → LLM sub-agents. Consolidation/learning jobs need three storm-proofing fields: cooldown_hours (timestamp written ONLY on successful runs), content-hash idempotency keys (reruns are no-ops), and per-run cost caps. Consumed queue items are deleted only after an error-free cycle (at-least-once semantics). See automation-and-triggers.

24. Skill self-tests and efficacy signals

Learned artifacts should declare their own measurement hooks: self_tests (one-line checks the agent applies to its own output, e.g. “every changed line traces to the request”) and efficacy_signals (“how to know it’s working” observables — diff-lines-per-request trend, clarifying-questions-before-first-edit rate) evaluated over the run ledger. Embedded eval_prompts (baseline prompt + rubric) and output_contract in a template give the evaluator a per-template rubric instead of a generic judge. Descriptions must answer WHEN, not WHAT — a description that summarizes the steps causes the agent to act on the summary and skip the body (lint for this at proposal time).

Patterns & compositions

The canonical flywheel shape. Multiple independent systems converge on the same pipeline: capture traces → summarize/judge (skip items already carrying authoritative scores) → aggregate evidence by artifact (loaded-only attribution) → decide (four-action space) → generate candidate → verify (reject-by-default gate) → validate (replay/held-out/statistical) → publish as proposal/overlay → attribute outcomes (predict-then-verify) → feed effectiveness back into retrieval.

Capability acquisition ladder. Solve once → make repeatable (capture trajectory) → skill → workflow (typed phases + checkpoints) → specialized harness (deterministic rails + gates) → eval coverage → automation → monitoring → trust-based autonomy → packaged reusable asset. “Most capable” = repeatedly absorbing new domains through this ladder, not one impressive run. Template lifecycle metadata should encode the current rung so the flywheel proposes the next rung, not generic improvements.

Graduated autonomy with evidence-gated promotion. L0 draft → L1 report-only (mandatory week one for any new pattern) → L2 assisted (small auto-fixes + separate verifier + attempt cap) → L3 unattended (requires denylist, budget file, run log, human gates, demonstrated activity). Promotion is earned from measured outcomes (triage accuracy at L1; two weeks of proven L2), tracked per skill/domain, not globally (“good at testing is not automatically good at deploys”); ramp for high-risk domains: shadow → recommend → draft-with-approval → bounded autonomy.

Capture at the API boundary for collective evolution. A local LLM-proxy that intercepts any agent’s model traffic records full session artifacts (turns, tool calls/errors, which artifacts were injected vs actually read/modified) agent-agnostically — the whole machine’s agent activity feeds one flywheel. Client and evolve-server couple ONLY through shared storage (no RPC); external library changes detected by mtime+size fingerprint + a generation counter; publish conflicts resolved by content-sha detection + LLM merge. Caution from production: synchronous storage calls inside the async proxy hot path cause stalls — keep persistence off the request path.

Evolver-as-agent in a jailed workspace. The stronger-but-costlier alternative to one-shot refiner calls: build a bounded workspace (read-only pre-summarized sessions, full artifact bundles, a manifest, and a written SOP), run an agent, detect changes by file-hash diff, upload changed bundles. The SOP mandates self-validation before finalizing (1-3 validation scenarios from the evidence, static checks, smallest safe smoke test, revert-if-unvalidatable) with the validation record written to the paired evidence file, and a hard file-access boundary. Suits periodic “deep curation” passes; even approved autonomous curation gets backup + diff + rollback.

One declarative artifact → eval suite AND production schedule. A single recipe/template compiles via two pure functions into both a runnable configuration and an evaluation suite for itself — so every template ships with the benchmark its own diffs are gated on. Pre-registration variant: for evaluation-bearing work, scaffold a domain spec (metrics, search/test splits, leakage risks, budget) BEFORE implementation.

External intelligence → bounded experiment. Scheduled monitoring of ecosystem sources with an ingestion filter, producing digest + ranked ideas; each item captured as {source, date, claim, relevance, confidence, suggested_experiment, status, outcome}; hard rule: “do not adopt any external claim into the core system without a local eval, shadow run, or replay-based validation.”

Momentum engine. Five live queues — now / next / blocked / improve / recurring — with mechanical anti-stall rules (same failure twice → add a guardrail/test, don’t retry and hope; no visible artifact progress → checkpoint + surface) and leading metrics: time from completion to next queued task, reusable assets per milestone, failures converted into evals/guardrails, % runs ending with explicit next actions.

Anti-patterns & failure modes

• Single-run LLM-judge acceptance. Statistically indistinguishable from noise (compounded by ±8-14% day-to-day provider variance); median-of-3 with an epsilon margin was the working fix.
• Self-graded benchmarks. An optimizer that generates its own eval and then optimizes against it games itself; the human-deleted bootstrap sentinel is the countermeasure.
• Blaming the artifact for agent failures. If the skill already contained the correct info and the agent ignored it, that’s an agent problem — bloating or rewriting the skill (especially deleting correct facts in favor of “go read the source”) makes it worse.
• Crediting injection instead of use. Grouping evolution evidence by what was surfaced into the prompt rather than what was actually read/loaded blames and credits artifacts that were merely visible.
• Editing without reading history. Refiners that don’t read prior version evidence oscillate — the named failure is “reverting past improvements.”
• Giant prompt surgery without eval protection. One bounded change per cycle, gated; everything else is untraceable.
• LLM-generated context files can hurt. A measured result: machine-generated agent-instruction files degraded performance while human-written ones helped ~4%, and agents burned 14-22% more reasoning tokens on context-file instructions — argues for hard token budgets on auto-surfaced context and fewer, failure-traceable entries.
• Stale-baseline feedback. Feeding the optimizer the original baseline eval report each round (instead of the latest) starves it of progress signal — a shipped bug in one working optimizer.
• Verifier theater. A verifier that doesn’t execute the check in isolation and report actual output signs off while CI fails; implementer and verifier must differ by agent/model/instructions, default stance REJECT.
• Ratchet-by-brainstorm. Rules added speculatively (not from observed failure) accrete into unfalsifiable prompt bloat with no retirement path.
• Beating “nothing” without a placebo. Proves only that more text helps, not that the content works.
• Instruction-based scope control. “Only edit X” as prompt text gets bypassed; enforce editable surface by deterministic diff against an allowlist (including the commit-first bypass re-check), optionally pushed down to an OS sandbox.
• Silent retry storms. Repeatedly failing self-improvement jobs must dead-letter with evidence and an explicit replay action, and a no-improvement halt (N=5) — not loop forever.
• Trigger words in pasted content. Auto-triggering artifacts from quoted history/examples causes accidental activation; fence quoted context out of trigger matching and test trigger precision with positive / explicit-invocation / pasted-history-negative / neighbor-domain-negative prompts.
• Junk-entity accretion. Auto-capture without a notability gate (“when in doubt, DON’T create — a junk page degrades search; a missing one can be added later”) degrades the corpus the flywheel learns from.

Quantitative findings

• Harness-only optimization with a frozen model: val score 0.560 → 0.780 (~40% relative) over 96 autonomous experiments in 18 batches; biggest single jump 0.64→0.78 from one rule strengthened to reference computed annotations.
• Self-growing regression suite: 0 → 17 cases over 18 batches; most late-stage candidates rejected — improvements became “genuinely additive.”
• Engine-owned completion verification lifted real completion 37.5% → 87.5% in one measured case; layered harness additions took another team 20%→60%→80%→~100%.
• Single-run LLM-judge acceptance ≈ noise; working fix: median-of-3 judge runs + epsilon 0.05 margin, plus a held-out gate with disjoint task IDs. Optimization cost ~$0.71-0.90 per 20-task run.
• Agentic proposer with filesystem access to all prior candidates beat compressed-text optimizers by +7.7 pts at 4x fewer context tokens (online classification) and +4.7 pts across 5 held-out models (math RAG).
• GateOK defaults: held-out subsample 50; other-cluster regression tolerance eps 1%; cycle acceptance min_improvement 0.02; stagnation stop k=5, eps 0.001.
• Publication-verifier acceptance threshold 0.75 (reject-by-default); generated-template acceptance vs one-pass baseline ≥0.80 on a 40/25/20/15 weighted rubric.
• Process-reward voting: M=3 parallel votes, temp 0.6 (capture), temp 0.1 (replay); session-judge dimension weights 0.55 task_completion / 0.30 response_quality / 0.05 efficiency / 0.10 tool_usage, overall recomputed server-side (never trust the judge’s self-reported overall).
• Effectiveness-blended retrieval: sim * (0.3 + 0.7 * effectiveness); near-duplicate prune at pairwise embedding sim > 0.9; stats flushed every 10 mutations.
• Push/surface confidence bases: alias 0.9, exact title 0.8, fuzzy 0.6, +0.05 recency bonus; gate 0.7; cap 3 (hard 5); events pruned at 90 days.
• Day-to-day provider variance ±8-14% (MoE/batching) — a 5% real change is invisible without a fixed canary suite.
• Failure mining auto-discovered 29+ distinct root-cause clusters without labeling.
• Abandonment ladder proven values: revert on gate fail; abandon hypothesis after 3 failures; halt after 5 consecutive non-improving iterations.
• Machine-generated context files hurt; human-written helped ~4%; agents burned 14-22% more reasoning tokens on context-file instructions.
• Version history caps: 20 registry entries; edit budget 4 edits/step (cosine schedule); ≥3 supporting evidence items before promoting a pattern/skill (two independent systems converge on the 3 threshold).
• Automated research ratchet (proven pattern): ~700 experiments in 2 days → ~20 stacked improvements → 11% training speedup, using a fixed 5-minute eval budget for comparability and only-improving commits.

Open questions

• N=1 replay validity. Fleet designs use distributed quorum over replay jobs; a single-user system gets one replay sample. How many local replays (or how much history) are needed before replay-vs-baseline beats a calibrated judge alone?
• Judge acceptance under provider drift. Median-of-3 defeats sampling noise, but ±8-14% day-to-day serving variance may still swamp small epsilon margins; is pinning snapshots sufficient, or do acceptance gates need drift-normalized scoring against a same-day canary?
• Rule retirement. The corpus is unanimous that ratcheted constraints must expire as models improve, and offers ablation as the test — but no system demonstrates an automated model-upgrade watchdog that re-benchmarks compensations at scale.
• Plateau exploration. Strict-improvement hill-climbing (both the monotonic ratchet and strictly-greater optimizers) never accepts sideways moves; frontier/Pareto search exists but no source quantifies when plateau exploration pays for itself.
• Self-referential closure. Can the optimizer safely optimize its own optimizer prompts (refiner, critic, judge)? Treating extractor/surfacing/curator prompts as first-class versioned tunable artifacts is proposed in the corpus, but no acceptance design prevents a degraded critic from approving its own degradation beyond the bootstrap sentinel.
• Placebo automation. The three-arm eval’s same-length placebo arm is load-bearing but hand-authored; nothing shows how to generate honest placebos automatically without leaking the treatment content.
• Cost of the full loop at personal scale. Individual mechanisms carry measured costs ($0.71-0.90/run, $5 preflight caps, max-usd remediation), but no source totals what a complete always-on flywheel costs per week on one user’s traffic, or which mechanisms to shed first under budget pressure.