Knowledge Pipelines

Part of the PersonalClaw research-learnings library. Source-agnostic; distilled 2026-07-13 from a 95-source competitive-research corpus.

Principles (the durable truths)

Compiled knowledge beats query-time retrieval for accumulating domains. Multiple independent systems converge on the same thesis: synthesize knowledge ONCE at ingest time into structured, cross-linked, typed pages — instead of re-deriving it per query from raw chunks (RAG). Contradictions get flagged at ingest, not discovered at answer time; cross-references are pre-built; each source permanently increases the store’s value. The knowledge base is “a persistent, compounding artifact.” The human’s job is curating sources, directing analysis, and thinking; the pipeline’s job is the bookkeeping — the maintenance burden that historically kills personal wikis drops to near zero, which is why the pattern works now.

The LLM is a formatter, never the source of truth. The strongest pipelines route every claim through a deterministic measurement or extraction layer, and use models only to organize, judge semantics, or link. Corollaries seen repeatedly: “scripts are the only writers; agents emit JSON”; extracted structured payloads are authoritative and rendered markdown is a presentation layer; computed/derivable fields (slugs, aggregate scores, IDs) are schema-forced to null in LLM output and computed by code.

Order work by cost, each layer shrinking the next layer’s input. Deterministic normalization/hashing (free) → bounded LLM passes on slim payloads → human review (scarce). This layering appears in entity resolution, dedup, lint/health checks, template matching, and fetch escalation — it is the single most-converged-upon economic shape in the corpus.

Structure is the contract; the intelligence tier is swappable. The same directory taxonomy + schema document can be driven by zero-model heuristics today and an agent tomorrow. A pipeline should degrade gracefully to a deterministic extraction floor when no model is reachable, stamping entries with extraction provenance so a background pass upgrades them later.

Idempotency and deterministic identity are what make pipelines re-runnable. Deterministic IDs, content-hash staleness keys, and idempotency keys covering ALL side effects of a write (body, counters, links, index entries) are the difference between an incremental pipeline and one that silently corrupts salience metadata on every re-run.

Provenance-per-claim is non-negotiable. Every extracted fact should carry a verbatim quote, a source reference, and (ideally) the author’s epistemic stance — so later judges, curators, and contradiction detectors can ground-truth it without re-reading the source.

Mechanisms (implementation-ready designs)

Claim schema with evidence and epistemic stance

Persist claims as structured objects, not free text: {statement, quote, source_ref, finding_type, hedging, confidence, fields[]} where:

• statement is phenomenon-level — no experiment-specific details (no dataset names/benchmark numbers); the numbers live in quote/source_ref (e.g. “single-head attention underperforms multi-head on MT quality”, not a BLEU delta).
• quote is the source’s actual wording, ≤200 chars; quotes stay in the source language even when names normalize to a canonical form.
• hedging ∈ {asserted, hedged, speculative} preserves the author’s confidence separately from extractor confidence (speculative claims can decay faster).
• finding_type ∈ {empirical, theoretical, definitional}. Quality rules: atomic (split compound claims), testable (future work must be able to support/contradict it), quote-over-paraphrase, no contributions-as-findings (“we propose X” → extract the underlying testable assertion). 3–8 claims per source is typical; 15+ signals over-splitting.

Deterministic identity & idempotent writes

• Entity IDs: {type_lower}:{unidecode_snake(name)} (e.g. person:harry_boyte); relation IDs src|TYPE|tgt|doc. Stable IDs make rebuilds reproducible and enable auto-merge across incremental runs.
• Idempotency keys must cover EVERY side effect: a documented bug class is deduping the content section but still incrementing a mention_count counter — re-ingesting the same source inflates salience forever. Key all mutations (body, counters, link records, index rows) on (source_id, target, op).
• Import/ingest exits early when the item already exists (keyed by canonical id / DOI / content hash); overwrite requires explicit consent.
• Consolidation jobs need three storm-proofing rules: cooldown_hours with the timestamp written ONLY on successful runs; idempotency keyed on (file_path, content_hash) so reruns are no-ops; per-run max_cost caps.

Stage-level caching with staleness reasons

Per-item extraction caches record {model_used, domain, chunk_size, extracted_at}; a staleness check re-extracts only when parameters changed, emitting human-readable reasons (“model changed”). Content-hash memoization (sha256(content) → cached edges/output) plus per-item JSONL checkpoints make expensive fan-outs resumable; unchanged inputs replay with zero API calls. Cache-key consistency is a real bug class: one system compared a truncated hash but stored the full hash, making refreshed pages permanently look stale.

Fixed per-item LLM-call budgets with role-tiered models

Declare “N calls per ingested item” as a design invariant. A proven paper-ingest shape uses exactly 4 calls: one reasoning-heavy drafter (summary + adversarial critique on the big model), plus extractor / metadata / linker on a haiku-class model — everything else (fetch, scan, assemble, citation match, stubs, edge application, lint) is 0-token code. Model tiering is inverted from intuition: the evaluative/critique call gets the expensive model; extraction gets the cheap one. “Per-item LLM loops are forbidden” — anything iterating an agent over a list gets refactored into a script over LLM-emitted JSON.

Purpose-cut input slices; full text never reaches an LLM

At fetch time, cut each document into role-sized slices: .brief (abstract/intro/conclusion, ~10–25% → summarizer), .findings (~40–60%, method+results, references stripped → claim extractor), .meta (first 2 pages → metadata), full text reserved ONLY for deterministic passes (citation matching). Token cost is controlled by input shaping, not prompt pleading. Cascaded deterministic section detection precedes any LLM: PDF outline/TOC cue-regex → font-size headings (body size = char-count-weighted mode; heading = >1.1× body and ≤120 chars) → page-header regex fallback; first 3 and last 2 pre-bibliography pages always kept. Sources cache under sha256 keys.

One combined extraction prompt per chunk + document-context injection

Extract entities AND relations in one JSON response (halves LLM calls per chunk). Entities: {name, entity_type, attributes, confidence 0-1, context} (verbatim quote = provenance); relations reference entities by NAME not ID, with evidence quotes. Anti-hallucination rules: extract only explicit information; never infer relationships from co-occurrence alone; empty relations allowed. One extra call per document summarizes the first chunk in 2–3 sentences and prepends it to every chunk prompt as “DOCUMENT CONTEXT” — a cheap fix for chunk-local blindness. Accumulate all unique per-entity context quotes across chunks (joined with a separator) so a later synthesis pass sees everything.

Discover-then-freeze schema

For unknown domains, sample first chunks of up to 5 docs (~3000 chars each) and ask the LLM as “a knowledge graph architect” for 5–15 entity types + 8–20 relation types (UPPERCASE_SNAKE_CASE, specific over generic, active voice, per-relation source/target type constraints); save the result with schema_free: false. The discovered schema becomes a fixed contract so later chunks and documents don’t drift — zero-config UX with cross-run stability. See planning-and-decomposition for the same shape applied to plan templates.

Layered entity resolution, cheapest first

• L1 deterministic pre-dedup (free): unidecode → lowercase → iterative title-prefix stripping (~35 prefixes) → per-word singularization → group by normalized form → embedding-hash fuzzy self-dedup at threshold 0.95. Canonical pick: most frequent → longest → alphabetical.
• L2 LLM merge proposals (bounded): entities grouped by type, batched in overlapping windows (batch 100, overlap 20, stride 80 — overlap eliminates cross-batch blind spots); persons sorted by surname key so “Mr. Edwards”/“Bradley Edwards” co-batch; optional embedding-KMeans batching (target cluster ~100) with graceful fallback. Send only identity-relevant attributes (role/title/aka), never full records. The LLM must separate duplicates (merge groups) from variants (parent/child → typed EXTENDS relations) — merges and links are different apply operations. Cross-type same-name dedup needs no LLM (canonical = highest-degree node).
• L3 human review (scarce): proposals persist as DRAFT/CONFIRMED/REJECTED records; re-running the proposer only ever ADDS drafts and NEVER revisits human decisions; skip keeps the item pending. auto_approve_threshold=0.85 on min-member-confidence (1.0 disables).
• L4 graph surgery: merge node data into canonical (max confidence, union sources, member attributes fill only missing keys, member names recorded as aliases), rewire edges through the merge map, drop self-loops; a rejected A→B relation also removes B→A.

Evidence-accumulating edges with corroboration confidence

Repeated mentions of the same (source|type|target) triple merge into ONE edge carrying a mentions list (doc, confidence, evidence quote each) plus support_count/support_documents. Confidence re-aggregates across mentions via product-complement 1 − ∏(1−cᵢ) — independent weak signals reinforce (alternatives: mean/max). Displayed evidence = highest-confidence mention. This is the concrete alternative to overwrite-or-decay for corroborated facts, and support_count becomes a retrieval-ranking signal.

Edge provenance taxonomy + typed relation semantics

Two independent designs converge on tagging link origin: EXTRACTED (deterministic wikilink/reference parse, confidence 1.0) vs INFERRED (LLM, ≥0.7) vs AMBIGUOUS (<0.7); dedup keeps the highest-confidence edge with EXTRACTED beating any inferred duplicate. A richer claim-level taxonomy: supports (directed, evidence-for), contradicts (bidirectional; “numeric results with non-overlapping intervals qualify”), extends (directed), uses (directed), similar-to (bidirectional; explicitly weaker than supports — same claim on different evidence is similar-to, no evidential link). Claim-level edges auto-aggregate to document-level (uses→builds-on; cites from deterministic reference matching); scripts own graph invariants (mirroring, self-loop skip, aggregation) — the LLM only proposes. ≤5 edges per new item, justification ≤25 words, targets only from a supplied candidate list, never link items from the same source.

Constant-cost linking (the economics contract)

“The 100th item costs the same as the 10th”: before the one linking LLM call, a deterministic pre-filter scores candidates (score = field_overlap × 2 + author_overlap, drop if both 0), sorts desc, caps at 30, and excludes same-source siblings; candidates carry only {slug, statement, fields}. Linker input is graph-size-independent by construction — the structural alternative to caching-based mitigation. Any cross-linking or memory-association pass needs this bound or it degrades as the store grows.

Deterministic-replaces-LLM where identifiers exist

A citation-matching cascade fully replaced an LLM citation agent: arXiv-ID regex (version-insensitive) → DOI regex (\b10\.\d{4,}/[^\s,;)]+) → fuzzy title (exact substring, then quick-ratio ≥0.5 gate before a sliding window advancing in half-title steps, match ≥0.85, early-exit ≥0.9) → first-author-surname + pub-year co-occurring in an 80-char window; bibliography located by a references|bibliography header. “The LLM never sees the reference list.” Generalize: URL/message-id/file-hash matching when linking ingested items to existing knowledge should be code, not model. Caution: thresholds belong in ONE constant — a verified doc-vs-code drift had the same window documented as 6, 60, and 80 chars.

Compiled truth + append-only timeline (the page shape)

Every entity page = current synthesized understanding at top + append-only dated evidence entries below. Consolidation rewrites ONLY the compiled section; evidence is never rewritten. This solves “memory either goes stale or grows unboundedly.” Source-precedence ladder for conflicts (high→low): user’s direct statements → compiled truth → timeline entries → external sources. Contradictions: note both claims with citations — never silently pick one. Companion pattern: a living overview document for the whole store, revised in place per ingest (distinct from the append-only ops log). Also: session/episode compaction — after N episodes, generate a summary node linked to constituents, tracking compression ratios.

Zero-LLM extraction floor + write-time graph wiring

• Heuristic extraction floor: lowercase/strip/split; unigrams counted if length ≥3 and not in a ~130-word stopword set; bigrams get a +2 boost per occurrence; cap 30 candidates; classify bigram-or-domain-keyword → entity, else concept; summary = first paragraph >20 chars truncated to 500. Crude, but the whole ingest runs offline at zero cost and produces a navigable linked store. Stamp extraction: heuristic|llm so an enrichment pass upgrades entries when models return.
• Write-time deterministic linking carries real retrieval weight: on every page write, three regexes (markdown links, wikilinks, typed-link blockquotes) + a fixed heuristic type cascade (FOUNDED → INVESTED → ADVISES → WORKS_AT → MENTIONS) create typed edges; a published ablation attributes a +31.4-point P@5 lift to this graph layer alone vs the same hybrid-retrieval stack graph-disabled — worth more than embedding/reranker tuning, at $0 in tokens. A 17K-page full extract completes in seconds via batch SQL insert.
• Bounded fan-out: from ≤30 candidates, link the top 15 per type and create pages only for the top 10 — explicit caps prevent one source exploding the store.

Contradiction flagging at ingest

On persist, retrieve semantically-near or graph-linked entries and check for conflict; store conflicts as first-class records (bidirectional contradicts edges + a dedicated “claims in tension” view — arguably the highest-value output of the whole graph investment). Structural detection in a reified graph: statements with same subject+predicate but different objects, or same subject+object but different predicates. Answers over conflicting claims must acknowledge both sides with citations. Anti-pattern: windowing contradiction detection to the 5 most-recent pages, or one shallow whole-corpus prompt — both miss most conflicts (acceptable only as the cheap tier).

Reified temporal statements

For stores that need time: statements (SPO triples) as first-class NODES, not edges, carrying temporal validity (validAt/invalidAt), an aspect classification (identity/preference/goal/decision/event/…), and provenance chains back to source episodes. Dual storage split: what the user SAYS (directives, preferences, beliefs) stays whole; what the pipeline OBSERVES decomposes into triples. A holder-attribution layer makes beliefs principled: each claim row has holder ∈ {world, brain, person:<id>, …} + weight in 0.05 increments; retweet-amplification caps at 0.55, self-reported facts get 0.75 not world-consensus; every take must pass the “so what” test (load-bearing for some future query). See memory-architectures.

Health/lint cost tiering with cadence rules

• Health (deterministic, zero-LLM, run first every session): stub detection (body <100 chars), index-vs-disk sync diffs, log coverage (every source page must have an ingest log entry). “Run health first — linting an empty file wastes tokens.”
• Lint (LLM, every 10–15 ingests — cadence keyed to mutation count, not wall-clock): deterministic layer (orphans with 0 inbound links; broken wikilinks; missing entity pages mentioned on ≥3 pages; sparse pages <2 outbound links) + graph-aware layer (hub-stubs: degree > mean+2σ but content <500 chars; fragile bridges: community pairs joined by exactly 1 edge; isolated communities = knowledge silos) + one cheap semantic prompt (contradictions/staleness/gaps/suggested sources).
• Scoped incremental lint: run auto-repair scoped to just-written items on every ingest (auto-wire near-duplicate similar-to at difflib ratio ≥0.92); whole-store lint is a separate sparse command.
• Graph health tiers: edges/node ≥2.0 healthy / ≥1.0 warning / else critical; orphan rate target <10%; “phantom hubs” = wikilink targets referenced by ≥2 pages that don’t exist (ranked by ref count) — each metric implies a concrete repair action (heal/expand/link). Lint should also emit research suggestions (data gaps fillable by web search, next questions), not just repairs.
• Post-write deterministic validation after EVERY LLM-driven ingest: created pages’ links resolve, page appears in the index — free structural verification before any judge runs (see verification-and-judging).
• Rule-based graph cleanup pass (no LLM, each step with dry_run): passive→active relation flipping (~21 explicit mappings + regex ^[A-Z]+ED_BY$ → strip _BY, flip edge), transitive-edge reduction only for declared transitive relations (e.g. LOCATED_IN), isolated-entity pruning, relation-type synonym normalization with a fallback type, direction fixing from schema type constraints.

Schema/conventions contract (co-evolving)

Three-layer ownership: (a) raw sources — immutable, pipeline reads but never modifies; (b) the wiki/store — pipeline-owned entirely, human reads; (c) the schema doc — page formats, naming conventions, ordered ingest workflow, validation criteria, all in one self-contained document that both deterministic scripts and agents follow. The key loop: learned per-user workflow preferences get written BACK into the schema “for future sessions,” not left in chat history. Declaring per-layer write ownership is a safety invariant; autonomous synthesis runs under write-scope allow-lists (path/prefix globs) so even prompt-injection success cannot write outside the declared scope — see security-and-guardrails. Uniform frontmatter (type: source|entity|concept|synthesis, title, tags, sources, last_updated) + wikilinks + index.md catalog + append-only grep-parseable log.md (## [YYYY-MM-DD] <op> | <title>, closed op vocabulary) is the de-facto interchange layout — at least four independent systems ship it, git-versioned and Obsidian-compatible.

Index-guided retrieval before embeddings

Maintain a one-line-summary catalog (title + summary + kind + date, by category) the LLM reads FIRST, then drills into chosen entries. Works “surprisingly well at moderate scale (~100 sources, hundreds of pages)”; a sibling design puts the ceiling at ~2,000 articles (~2M tokens) before hybrid retrieval becomes necessary as a pre-filter. Two-stage query retrieval when lexical match is thin: keyword title-match + one-hop graph expansion over edges with confidence ≥0.7, overview always injected at position 0, capped at 15 pages; LLM path-selection fallback only if ≤1 match. Every answer cites wikilinks; conflicting findings acknowledged both-sides; “never fabricate a citation — if no evidence exists, say so and suggest a source to ingest.” Query→persist flywheel: good answers file back as first-class synthesis entries with question/consulted-sources/date frontmatter — explorations compound like ingested sources. Reserve the synthesis/analyses taxonomy slot BEFORE the query feature exists so the whole system already handles filed answers.

Communities, topology, and narrative

Louvain community detection over the cleaned graph (seed it — unseeded Louvain is nondeterministic; one system uses seed=42, another shipped the bug) with min community size 8; ONE LLM call names all community themes ($0.01 to regenerate labels). A compact topology JSON (communities with top entities, bridges with cross-community edge counts, isolated nodes) is the agent’s session-start orientation map; the highest-value reasoning pattern is “link knowledge islands” — find community pairs with near-zero shared edges and name specific entities worth connecting. Snapshot topology before/after an ingestion wave and report the diff (new clusters/merges/bridges) as the synthesis output, not just “N items added.” Narrative generation: overview from top-50-by-degree entities; key-connection chains via shortest paths (2–4 hops) between top entities; entity descriptions only for degree ≥3, capped, async under a semaphore with running cost display; a banned-phrase regex list (“played a key role”…) triggers a constrained rewrite (“keep everything else EXACTLY the same”).

Extraction-quality self-reporting

Every automated extraction run should emit a meta manifest: coverage stats (items read/blocked/recovered with failure causes broken down: ssrf/http/timeout), which fields hit collection caps, partial-data warnings. Downstream judges/curators weigh evidence by this envelope. Pair with per-section inference-honesty markers in the rendered output: “Note: these values are inferred defaults, not measurements from the source” — measured vs inferred visually distinguishable per section.

Feed detection & normalization (manufacturing sources)

Turn any web page into a structured item stream with five parallel zero-LLM detectors, all enabled by default: (1) platform-API detection (e.g. a WP REST link tag → pull posts via API instead of scraping), (2) JSON-LD/Schema.org blobs, (3) semantic HTML5 (<article>/<main>/<section>), (4) structural frequency analysis (frequently occurring selectors likely to contain main content; tunables minimum_selector_frequency: 2, use_top_selectors: 5), (5) SPA state blobs (__NEXT_DATA__, __NUXT__, window.STATE — walk for arrays with title/url pairs). Declarative selector config for the failure cases: item selector + per-field extractor (text/html/href/attribute/static) + post_process chain (gsub, html↔markdown, parse_time, relative-URL resolution, sanitize_html “always recommended”, template). Hygiene knobs: drop off-domain items (kills ad/recommendation links), min_words_title: 3; a valid item needs at least title or description. Escalating fetch chain with one shared budget: plain HTTP → scrape API → real browser, where escalation is driven by extraction outcome (did we get items?) not HTTP status, all attempts drawing on one max_requests budget, escalations logged. Novelty gating: every item needs a stable guid (composable from extracted fields) + a persistent seen-set, or monitoring loops re-process the same items forever. Failure-diagnosis UX: “change the input URL first before assuming setup is broken” — listing/changelog/archive pages work; homepages and single posts don’t. Filters compose: a saved query over streams is itself re-exportable as a new stream. Community-shared extraction recipes per site are the template idea applied to sources.

Report/document/infographic rendering from specs

• One source, N presentations: a doctype switch (plain | paged | slides | docs) over the same markdown source, with orthogonal color×layout themes — presentation is a render-time parameter, not a different artifact. Media storage copies referenced files into the output with content-hash suffixes (img@HASH.png) and rewrites paths so exports are self-contained.
• Declarative infographic DSL designed for LLM authorship: split data (title/desc/items in 6 canonical shapes: lists/sequences/compares/values/hierarchy/relations, exactly ONE primary field) from design (structure + item templates) from theme. A template is just a registered design preset — pre-define design, leave theme open. Fault tolerance IS the feature set: never-throw line parser accumulating typed issue records {path, line, code, message, raw} on a warnings channel; lazy array promotion (containers start as objects, retroactively become arrays); markdown-fence stripping; fuzzy template-name resolution (Levenshtein, no cutoff — a slightly-wrong name always lands on something renderable); union schema mapping keeps the variant with fewest errors. Progressive streaming render: append chunk → re-parse whole buffer → full re-render with a staleness guard on async completions. getTypes() derives a machine-readable data contract from a template’s declared composites and hands it to the LLM before parameter-filling. Same spec renders three ways: interactive+editable, streaming, headless SSR → standalone SVG with embedded fonts.
• Governed generative HTML documents: templates lock style (exact-hex palettes with usage domains, a fixed pool of ~22 composable layouts, “iron laws”: border-radius 0, hairlines, grid) while explicitly freeing quantity to be content-driven (“template ranges are a reference floor, not a ceiling”; every segment of user content must be covered). Shared hard directives claim override authority over any template number. Anti-slop rules: 1 primary + 2 neutrals + ≤1 accent, 8px baseline grid, 65ch measure, contrast ≥4.5, real user data only — structured data must yield charts/tables of actual insights, never lorem ipsum. Artifact contract: stream the artifact as the entire reply body between exact sentinels (<!DOCTYPE html>…</html>), file tools forbidden — plus a rescue path that lifts the artifact from tool_use arguments when the model writes a file anyway. Lenient multi-strategy extraction with a guaranteed-render scaffold fallback (escape raw text into a <pre>) so a preview is never blank; close unbalanced tags for mid-stream preview.
• Dual-artifact output everywhere: authoritative structured JSON + presentational markdown/HTML, both persisted and linked; the interesting periodic output is the diff against the previous version-tagged snapshot, not the snapshot itself.
• Compressed agent-facing digest: every reusable spec ends with a deliberately compressed restatement (quick reference + ~10 numbered rules) for cheap context injection — a two-tier context budget distinct from the full document.

Ingest pipeline shape (reference)

The converged 10-step ordered ingest: read source (auto-convert non-markdown) → read index+overview for context → write source page (fixed sections: Summary 2–4 sentences / Key Claims / Key Quotes / Connections with relationship notes / Contradictions; domain templates override sections by source type) → add index entry → revise living overview if warranted → create/update entity pages → create/update concept pages → flag contradictions → append log entry → deterministic post-ingest validation + change summary. A single source may touch 10–15 pages; caps keep it bounded. Conversation-sourced variant: cheap per-session flush (no tools, 2 turns, extraction taxonomy: Context / Key Exchanges / Decisions Made with rationale / Lessons Learned / Action Items; explicit FLUSH_OK no-op sentinel still logged so absence of memory is observable) → immutable daily log → expensive batched compile pass that sees the whole corpus and is instructed “update existing articles rather than creating near-duplicates” (dedup-by-showing-the-corpus works to ~2k entries). Cron-less piggyback scheduling: after each flush, if hour ≥ H and today’s hash differs from last-compiled, spawn the compiler detached.

Patterns & compositions

• Fixed pipeline of resumable, cached stages — init → extract → build → resolve → review → apply → narrate → export — with cheap deterministic stages separated from LLM stages and from human stages; full automation deliberately stops where judgment is needed (resolve/apply never auto-run). Maps onto workflow engines as ingest→foreach-extract→transform→action→human-gate→action; see workflow-engine-design.
• Accumulate → synthesize → persist: accumulate per-item contexts/observations across chunks or sessions into a staging tier (append-only daily logs, " ||| "-joined context lists), then run one synthesis over the accumulated material. The staging tier decouples capture latency (instant, I/O-only) from synthesis quality (slow, batched); compiled entities cite staging entries back.
• Documents as first-class graph nodes with MENTIONED_IN provenance edges to every entity, plus a strip-metadata “substantive” view for analytics/communities — provenance queryable but never polluting similarity structure.
• Projections are full rebuilds from truth: regenerate the index/catalog from disk state every run rather than patching incrementally — drift becomes impossible by construction instead of detected by a health check. Where incremental is kept, a health check must diff index-vs-disk.
• Dangling links as growth affordances: render references to not-yet-existing entries visibly (❓ marker) in the reading UI; click offers “draft this entry” (through a proposal queue). Phantom-hub detection (≥2 refs, no page) is the batch version; gap-healing drafts the page from ≤15 source excerpts — but should emit proposals, not direct writes.
• Ingestion-rules per source: user-configurable extraction shaping per connected source (“from email, only extract action items and decisions; skip newsletters”).
• Refresh via source back-pointers: ingested entries carry source_file + hash frontmatter; a refresh pass re-ingests only entries whose raw source changed.
• Query grammar before LLM triage: a terse deterministic operator language (intitle:, author:, #tag, ISO-8601 date:P1M, regex, boolean+parens) does 90% of stream triage with zero tokens; human raw-pass-through mode must stay first-class (some users explicitly reject LLM curation).
• Delta-seeking enrichment: send external searches what the store already knows so they return only the DELTA; effort scales to entity importance (3-tier: full research / web+cross-ref / cross-ref only); brain-first lookup before creating anything; a notability gate (“when in doubt, DON’T create — a junk page degrades search; a missing page can always be added later”).
• Knowledge-store site export: links define a build graph; per-page outputs share theme assets; orphans skipped; collision-proof naming (name@hash) — a folder of pages becomes a browsable static wiki.

Anti-patterns & failure modes

• Computing an idempotency hash but never checking it — one nightly automation re-ingested everything every night because ingest computed sha256 and never compared it.
• Partial idempotency — deduping the content write but still incrementing counters; salience metadata silently corrupts on every replay.
• Truncated-vs-full hash cache mismatch — permanent false-stale; cache keys must be produced and compared by the same function.
• Direct-write gap healing / auto-linking without review — LLM-drafted pages and semantic edges written with no proposal step; policy that works: deterministic reversible link-writes may bypass review, semantic or destructive mutations (merges, deletions, rewrites) never do.
• Contradiction detection windowed to recent items (5-page window) or one shallow whole-corpus prompt — structurally blind; use graph-linked/semantic neighborhoods at persist time.
• Trusting the LLM to emit identifiers directly — template IDs, slugs, entity IDs. Working fix: LLM output re-enters a deterministic scorer/resolver (summarize-then-rematch; fuzzy resolution with warnings); scripts compute slugs.
• Unbounded linking cost — per-page LLM inference over the whole store grows O(n); cap candidates deterministically instead.
• Threshold drift between docs and code — the same constant documented as three different values; single named constants.
• Junk-entity accretion — auto-capture without a notability gate degrades retrieval permanently.
• Unseeded community detection — nondeterministic clusters break caching and diffing.
• O(n²) lint with repeated file re-reads — fine at prototype scale, an anti-pattern beyond it; build the link map once.
• Silent no-ops — a memory/ingest pass that saves nothing must record that fact (FLUSH_OK); a week of all-no-ops on an active system means broken capture, and without the sentinel it looks identical to health.
• Incremental index patching without a drift check — the catalog diverges from disk; either full-rebuild or health-diff.
• Claiming capabilities docs-first — a README advertising “contradictions flagged on ingest” while the status section admits it isn’t built; keep an explicit machine-auditable “already real vs still missing” split.

Quantitative findings

• Write-time deterministic typed-edge graph: +31.4 points P@5 vs the identical hybrid-retrieval stack with the graph arm disabled (P@5 49.1% / R@5 97.9% on a 240-page corpus; ripgrep-BM25, vector-only, and hybrid+RRF-no-graph all ~18); ingestion costs $0 in tokens; 17K-page full extract in seconds.
• Fixed ingest budget: exactly 4 LLM calls per paper (1 reasoning-class + 3 fast-class); linking capped at ≤30 candidates keeps marginal cost flat as the graph grows.
• KG extraction cost example: 425 entities ≈ $0.72; community re-labeling ≈ $0.01; conversation-memory ops: flush $0.02–0.05/session, compile $0.45–0.65/log, query $0.15–0.25, structural lint free.
• Combined entity+relation prompt cuts LLM calls per chunk from 2 to 1.
• Dedup batching: window 100, overlap 20 (stride 80); embedding-hash pre-dedup threshold 0.95; auto-approve threshold 0.85 on min-member confidence; auto-wire near-duplicate similar-to at difflib ≥0.92; edge-confidence split INFERRED ≥0.7 / AMBIGUOUS <0.7.
• Index-guided retrieval viable to ~100 sources / hundreds of pages (one system) and 50–500 articles fine / ~2,000-article (~2M token) ceiling (another) before embedding pre-filters are needed.
• Lint cadence: deterministic health every session; LLM lint every 10–15 ingests. Graph health: edges/node ≥2.0 healthy / ≥1.0 warning; orphan target <10%; god-node = degree > mean+2σ; hub-stub = high degree + <500 chars; missing-entity trigger = mentioned on ≥3 pages; pattern-page promotion threshold = ≥3 supporting reflections (independently converged with “≥3 lessons → skill proposal”).
• Claim extraction: 3–8 findings/source typical, 15+ = over-splitting; quotes ≤200 chars; ≤5 edges per new item, link justification ≤25 words.
• Skill/context grounding for a spec-emitting agent: with bundled offline docs + strict-compile verify loop, 4/5 first-try compiles, 6 total attempts, 0 silent spec misses, 92% quality vs 0/5, 14 attempts, 3 silent misses, 63% without — at +58% tokens but flat wall time (see skills-and-prompt-craft).
• Section detection heuristic: heading = font size >1.1× body (body = char-count-weighted mode) and ≤120 chars; brief slice ~10–25% of doc, findings slice ~40–60%.
• Fan-out caps: 30 extraction candidates → 15 linked → 10 pages per source; retrieval page cap 15; heal context ≤15 pages × 800 chars; stub = body <100 chars; sparse = <200 words or <2 outbound links.
• Feed synthesis tunables: selector frequency ≥2, top 5 selectors, min title words 3, pagination cap 10 pages; a memory-store benchmark using reified temporal statements reports 88.24% average accuracy across single-hop/multi-hop/open-domain/temporal reasoning.

Open questions

• Where exactly does index-guided retrieval stop scaling? The corpus offers ~500-page and ~2,000-article heuristics from different systems; no measured crossover against hybrid retrieval exists.
• Paraphrase-level dedup: every system in the corpus punts (string similarity at 0.92, or show-the-whole-corpus); embedding-based claim dedup with proposal review is unproven at personal scale.
• Contradiction detection recall: persist-time neighborhood checks are cheap but their miss rate vs full pairwise comparison is unmeasured; nobody has published numbers.
• Temporal reasoning as first-class retrieval: “what was true last week but isn’t now?” — reified validAt/invalidAt statements exist, but no system demonstrates multi-hop temporal queries working well; noted as a known weakness even by the strongest graph system.
• Round-tripping manual edits into declarative specs: post-generation visual editors exist, but whether hand edits write back into the source DSL is undocumented in every rendering system examined.
• Corroboration model calibration: product-complement 1−∏(1−c) treats mentions as independent — same-author or same-upstream-source mentions aren’t; no system corrects for provenance correlation.
• When to auto-switch extraction tiers: the heuristic-floor → LLM-enrichment upgrade path is designed but no system reports measured quality deltas or the right trigger (provider-available event vs sparse cadence vs backlog size).