Local Models & Inference

Part of the PersonalClaw research-learnings library. Source-agnostic; distilled 2026-07-13 from a 95-source competitive-research corpus.

Principles (the durable truths)

Local models already cover most personal-assistant work; the bottleneck is the operations stack, not the weights. Measured research puts local models at 88.7% success on single-turn chat/reasoning queries. Every mature local-first system converges on the same missing layer: download lifecycle, capability declaration, process supervision, hardware fit, and honest failure surfaces. The model catalog is a solved problem; model operations is where products differentiate.

Capability must be declared as structured data, never discovered at failure time. Engines/models that carry explicit capability flags (supports_cloning, structured_output: json_schema, speaker_labels, hotword_budget) let callers gate features and dispatch strategies; systems that infer capability by trying and catching produce silent fallbacks and dead paths. Multiple independent systems converge on this: engine ABCs with capability class-attributes, per-backend structured-output feature matrices, and per-model I/O contracts stored in the catalog.

Probe by execution, not assumption. The only trustworthy evidence that a backend/binary/model works is running it: execute the binary with a device-listing flag, run a tiny real synthesis as a self-test, benchmark each accelerator with a real completion. Filesystem presence, file extension (a .gguf is not evidence the runtime supports the architecture), and manifest claims all lie.

Native inference libraries must be process-isolated. Torch/faiss/av/loky-class native code segfaults in-process and takes the host gateway down with it. Sidecar subprocesses with dedicated venvs, child-reported memory, and supervised lifecycles are the structural fix; environment-variable band-aids (thread pinning, tokenizer parallelism flags) only shrink the window.

Constrained generation makes the parser the generator. Where you own the logits, masking invalid tokens per-step guarantees output shape at near-zero overhead and measurably improves accuracy and prompt-robustness; where you don’t, degrade explicitly to provider schema modes or parse-with-retry. Structure is scaffolding for reasoning, not just formatting — but only if the schema leaves reasoning room.

Hardware awareness should drive defaults, honestly and outside the binding path. Tiering (RAM/VRAM), quant preference ladders, and fit estimation with human-readable reasons produce a “one right file per repo” UX. But fit verdicts belong in search/recommendation surfaces; hard-blocking bindings on fit estimates causes false negatives as estimation drifts.

Efficiency is a first-class run metric. Per-step tokens, TTFT, latency, energy (joules/watts), and GPU utilization recorded on every inference step is what makes learned local-vs-cloud routing, Pareto optimization, and cost dashboards possible later. Systems that skip telemetry can never grow the routing layer.

Degrade with machine-readable reasons; complete rather than fail. Pipelines missing an optional capability (no token, gated license, no GPU) should finish degraded with exact reason codes (diarization_skipped:no_token|401|network) plus a docs deep-link — never a hard failure, never an unexplained silent fallback.

Mechanisms (implementation-ready designs)

1. Engine ABC with capability flags and message-carrying availability

A pluggable engine base class: class attrs id, display_name; abstract sample_rate, supported_languages; classmethod is_available() -> tuple[bool, str] where the message explains why not (and a (True, "ready — <advice>") convention carries upgrade hints); a uniform generate(...) signature with keyword extras. Non-abstract capability flags gate features instead of silent fallbacks: supports_cloning, supports_voice_design, applies_own_mastering (skip shared post-processing), gpu_compat (advisory only). Lifecycle: ensure_ready() separates load budget from generate budget; idempotent unload() clears attributes enumerated in a _MODEL_ATTRS list (which also powers loaded-model detection). Registry is lazy (id → (module_path, attr) resolved on first access) to break import cycles; availability probes mask secrets in messages; switching active engines unloads the outgoing one.

2. Sidecar subprocess isolation with resumable install jobs

Heavy engines run as subprocesses in dedicated venvs. Install is a background job polled via GET .../install/status returning {engine_id, installed, managed, install_dir, job: {state, steps[], log[], error, remediation, weights_progress}} — the remediation field (actionable next step, distinct from error) is the key detail. DELETE refuses user-managed installs and 409s while a job runs. Health for sidecars = spawn + ping; in-process = is_available(). Route design gotcha: keep sidecar install routes off dynamic path segments that would shadow literal routes. Child processes self-report VRAM so the observability panel attributes memory correctly.

3. Declarative model catalog carrying the runtime contract

One YAML/JSON source of truth per known model: repo_id, label, role, size_gb (expected size → truncated-download detector), required, platforms (e.g. darwin-arm64, cuda), note (UI tooltip), config_only (escape hatch for pipeline repos with no local weights so the truncation detector doesn’t misflag), engine-specific passthrough fields. Extend entries with the I/O contract as data: input tensor name, expected shape, value range, output semantics — this is what lets N heterogeneous models run through one pipeline instead of key.includes("modelname") dispatch. Add a versioned cache key (name_v2): bump it and the old artifact is orphaned and the new one downloads; “is this downloaded and current” becomes one comparison. Surface license + author at point-of-choice in the picker (including non-commercial warnings) — provenance as UX. Download-detection must probe every path a download/load can write (hub cache layouts differ from save layouts); delete must clear all layouts.

4. Download UX engineering

Convergent stack across multiple systems: (a) stream to <dest>.part, atomic rename on completion, leftover partials surfaced as cleanup candidates; (b) never report 100% until the artifact is persisted — cap at 99 during transfer; (c) stall timeout distinct from total timeout (e.g. 30 s between chunks → actionable “download stalled” vs 10 min overall); (d) three progress phases: Resolving (pre-flight exact total incl. cached bytes), Downloading (one bar with file count + speed/ETA), Done landing exactly on total; (e) segmented parallel byte-range download with automatic fallback to plain download on any error; (f) mirror endpoint probing in parallel, cache the winner, re-test after failures/7 days, never auto-switch an explicit pin; (g) gated-repo errors translated via the provider’s error headers (x-error-code: GatedRepo) into a concrete user action (“open page, accept license, add token”); (h) server-owned single canonical progress record with client reattach — the client owns no download state; on mount it polls the progress endpoint and re-adopts any running download; (i) companion-file auto-resolution: score dependent artifacts (vision projectors, tokenizer sidecars) by name-stem match (+10) and precision tier (bf16 +6, f16 +5), queue as automatic second download. Disable transport layers that bypass byte-level progress hooks (or accept losing honest progress).

5. Token/credential cascade with live validation

Three-source cascade: app-encrypted store (key derived per-install from machine id — document that copying the data dir breaks decryption and falls through) > env var > CLI token file. Rule: “the first source that has a token and survives a live whoami call wins” — invalid higher-priority tokens are skipped, not blocking. UI shows per-source masked preview, validated username, and an “Active” badge.

6. Hardware tiering, quant ladders, and fit estimation

Tier from a hardware snapshot: unified-memory machines by RAM (≥16 GB high, ≥8 mid), discrete by VRAM (≥12 GB high, ≥6 mid). Tier selects default search terms (“1B/7B/8B instruct”), context presets (2048/4096/8192), cache quant (q4_0/q8_0), batch (256/512/1024), GPU layers (0/-1/-1). Quant auto-selection ladder inside a repo: filter .gguf, exclude projector files and multi-part shards, walk a regex preference ladder — high tier q6_k → q5_k_m → q4_k_m → q8_0; low/mid q4_k_m → q4_k_s → q4_0 → q3_k_m → iq4_xs; first match wins so the user never picks a quant file. Fit estimation with honest reasons: runtimeNeed = sizeBytes × 1.25 + 1.5 GB; system reserve max(4 GB, 30% RAM); classify gpu / unified / hybrid / ram / too-large, each with a human-readable reason string. A richer variant: fit verdict by budget ratio (≤0.50 perfect, ≤0.78 good, else marginal) or discrete-GPU headroom (≥1.5× perfect); speed as bandwidth-bound raw_tps = memory_bandwidth / model_gb × 0.55 with a harmonic blend for CPU offload (cpu_bw ≈ 55 GB/s); composite score = use-case-weighted quality/speed/fit/context (e.g. general = 0.45/0.30/0.15/0.10); non-fitting models shown as no_fit, not hidden. Context can be halved down to 1024 before declaring no-fit; GGUF gets single-GPU VRAM (can’t shard) while AWQ/GPTQ/FP8 get the multi-GPU pool.

7. Backend probing and per-backend benchmarking with persisted winners

Validate accelerator backends by running each binary with a device-listing flag (10 s timeout, 60 s cache), parse per-device VRAM from output, filter virtual adapters by name, rank per-OS (cuda 0, hip 1, vulkan 2, sycl 3 … cpu 9). Then benchmark: restart the server per backend at ctx ≤ 2048, run one 64-token completion, read predicted tokens/sec, append to a capped newest-first ring buffer (max 100), and write the winner into per-model persisted settings {preferredBackend, benchmarkWinner: {backendMode, predicted_per_second, createdAt}} — empirical measurement becomes a durable preference with provenance, automatically. This “measure, then remember with evidence” loop generalizes to any calibration (see self-improvement-loops).

8. VRAM pre-flight and OOM degrade-ladder retry

Pre-flight before load: modelGB + 0.8 GB safety vs free VRAM; over budget → a 3-way modal (Cancel / Load on CPU / Proceed on GPU anyway). GPU-layer auto-tune estimates layer count from file size, scales usable VRAM by KV-cache quant multiplier (q4 = 0.25×, q8 = 0.5×, f16 = 1×), 85% headroom, all-layers when everything fits. On runtime OOM: classify by substring set (“out of memory”, “failed to allocate”, “failed to create context”), then walk a declared context ladder [32768, 24576, 16384, 12288, 8192, 4096, 2048, 1024, 512] to the next value below current, restart the engine, replay the request once (retry guard = 1). The general shape — classify error → apply next patch from a declared parameter ladder → re-execute once — is a reusable recoverable-failure policy for workflow-engine-design.

9. Resource arbitration for heavy runtimes

Complete small vocabulary, convergent across systems: (a) mutual exclusion with structured refusal — conflicting engine loads return 409 + code: MODEL_ALREADY_ACTIVE and the UI offers “Unload / Unload-and-Load”, never opaque failure; (b) per-engine promise/operation queues serialize all engine ops; (c) process-generation counters — every health-wait/completion handler carries the generation it awaited and aborts when superseded, killing the stale-server-squatting-the-port class; (d) identity-verified readiness — readiness polls must verify which model the port serves (match served model name against expected), not just 200 OK; (e) Semaphore(1) for model-backed background work with non-model housekeeping bypassing the slot, and the queued run-record written before the semaphore wait so UIs show “queued” truthfully; (f) interactive gate — background jobs wait for ≥1.5 s of no in-flight interactive HTTP requests, no browser heartbeat within 45 s, AND no live model stream; a running background task is cancelled and deferred (~15 min) when the user becomes active; passive endpoints (heartbeats, polls) excluded so polling can’t hold the gate; (g) pause GPU telemetry polling (nvidia-smi) while heavy image/diffusion inference runs — driver queries contend and cause display lag. See automation-and-triggers for the substrate-level slot/policy generalization.

10. Loaded-model observability and real self-tests

A “what is occupying my machine” endpoint enumerating every RAM/VRAM occupant across all sources: in-process engines (detected via non-None _MODEL_ATTRS), co-loaded pipeline models (marked unloadable: false), sidecars with child-reported VRAM, warm singletons with idle-timeout release — plus an is_active_engine attribution flag because models outlive engine switches, and a system RAM/VRAM pressure snapshot in the same payload. Targeted unload endpoints per occupant class. Health endpoints never 500 — exceptions become ok=False, message with secrets masked; latency included. Self-test endpoints run a tiny real inference of a fixed phrase, bounded (~90 s), serialized behind a process lock, user-initiated only, returning {duration_ms, sample_rate, num_samples, timed_out}.

11. FSM-indexed constrained decoding

The core algorithm: compile the target structure (regex; JSON schema → regex; CFG → pushdown automaton) into a finite-state machine, then precompute an index mapping every FSM state → the set of vocabulary tokens that are valid transitions. At decode time the valid-token mask is a lookup, not a per-token vocabulary scan — near-zero overhead per step; invalid tokens set to -inf before sampling; when the FSM reaches an accepting state only EOS is allowed. Embeddable primitives: Vocabulary (token→ID map + EOS, loadable from a tokenizer), Index(regex, vocab), Guide (stateful walker: get_tokens() / advance(token_id) / is_finished()), plus standalone JSON-schema→regex conversion. Compilation is expensive (FSM index over a ~100k vocab) — compile once per (schema, model) pair, cache the compiled processor, reset() state per call. A single type-compilation pipeline handles everything: Python type → term tree (Regex | JsonSchema | CFG | String | Sequence | Alternatives | KleeneStar | Optional) → regex → logits processor; int/float/bool map to predefined regexes, Pydantic/dataclass/TypedDict → JSON schema, Enum/Literal → alternatives; a plain function signature can be the schema (parameter names/hints → arguments extraction).

12. Structured-output capability dispatch (steerable vs black-box)

Two regimes behind one interface: steerable models (transformers/llama.cpp/MLX — you own the logits) support full regex/CFG/JSON-schema constraint; black-box API models degrade to whatever the provider supports (JSON-schema format param, response_format, or nothing). Make this a declared per-model capability matrix — structured_output: none | json_mode | json_schema | regex | cfg — and dispatch enforcement mode on it: constrained (guaranteed) / json_mode + validate / parse_retry (retry must re-present the schema AND the parse error location; models fail structure mostly when the schema wasn’t visible). Constraint backends themselves are pluggable behind one protocol (three interchangeable engines with per-constraint-type defaults). Design rules proven by measurement: put a bounded free-text reasoning field before the constrained answer field (constraints without reasoning room measurably hurt; with it they help); give a typed escape hatch (Union[Answer, Literal["cannot_answer"]]) so refusal is parseable, not a parse failure; show the schema in the prompt and mechanically lint that few-shot examples validate against the generation schema; validate a draft schema against a real data example before ever generating. Judge verdicts, enum-shaped triage decisions, and planner outputs are the highest-value call sites — see verification-and-judging and planning-and-decomposition.

13. Voice profiles as an entity, with lock-from-history and consent provenance

Promote voice from a settings string to a first-class entity: kind: clone | design; clone stores ref_audio_path/ref_text; design stores category picks (gender/age/accent/pitch/speed/emotion) + sanitized instruct; plus language, seed. Lock flow: promote a liked generation from history into the pinned voice (copy its audio to a locked file + pin the seed) — “freeze the voice I liked” beats parameter fiddling. Consent columns (verified_own_voice, consent text/audio/timestamp) are provenance, not biometrics: agentic features and sharing gate on the flag; plain local synthesis never does. Per-client voice bindings at the API/MCP boundary: agents self-identify via a client-id header; precedence = explicit profile arg > client binding > global default > built-in. Portable persona bundles: ZIP with integer schema_version, manifest + legacy shadow manifest for old importers, forced watermark on export (honestly recorded watermarked=false if the watermarker is absent), fresh server-generated IDs on import (zip-slip defense), non-forgeable verification (badge requires the actual consent recording ≥1000 bytes + non-empty text; manifest flags are advisory), best-effort forward import of future schema versions, mid-import rollback. Cloning quality envelope: 3–10 s reference clip sweet spot at ≥ −15 dB; zero-shot cloning from a 3 s clip is production-real.

14. Duplex voice-loop hardening

The working discipline for full-duplex local voice: (a) confirmation-phrase gating — accumulate streaming transcript and only fire the query on an execution phrase (“do it”, “go ahead”), with cancel phrases; push-to-execute semantics without a hotword model; (b) two-layer echo suppression — TTS start/stop callbacks mute the recognizer (drain queue + reset), AND drop any transcript sharing ≥3 consecutive words with the last TTS output (checked both directions); (c) pre-TTS text cleaning — strip inline code, reduce URLs to domain and paths to filename, drop CLI flags, trim explanation paragraphs to first sentence; (d) append a “transcription may be inaccurate” disclaimer to voice-originated prompts so downstream models self-correct; (e) speaking-interrupt support via a socket event. STT capture: stream mic → 16 kHz mono PCM16; streaming recognizers (Kaldi-class) give partials, grammar constraints, and word timestamps locally.

15. Joint transcribe+diarize and the audio capability matrix

A 0.9B Apache-2.0 model now does transcription + diarization + timestamps + acoustic events in ONE pass with float32 CPU fallback, beating much larger closed models on meeting cpCER — collapsing the classic three-part pipeline (STT ∥ diarizer → fusion node). Canonical output grammar [start][Sxx]text[end] (seconds; anonymous per-audio speaker labels) with a shipped parser → {start, end, speaker, text} segments; long audio handled by raising max_new_tokens (5120 → 65536). Consequence for provider contracts: coarse capability lists (["stt","diarization"]) are insufficient — models need a structured capability matrix: word_timestamps, segment_timestamps, speaker_labels, acoustic_events, hotword_biasing + hotword_budget, languages, max_audio_hint, time_aware_qa. A provider filling speaker at the source should declare it so downstream fusion no-ops. Hotword budgets are per-provider: one ASR family overflows a 224-token prompt window (overflow → silent empty transcript; cap ~200 chars), another takes hotwords as an instruction-prompt suffix that composes with custom instructions — never assume one model’s biasing limits.

16. Audio understanding models: time-markers and thinking budgets

Unified audio-LMs (audio encoder at 12.5 Hz → adapter → LLM decoder) subsume the pipeline: ASR, captioning, event detection, speaker/emotion analysis, and audio QA are all text prompts against one set of weights. Time-marker tokens inserted between audio frames at fixed intervals during pretraining make timestamp ASR and “what was said at 3:20” QA fall out of ordinary decoding — a ~6–20× advantage on temporal-grounding benchmarks over models with no explicit time tokens — togglable per-request via a processor flag. Cross-layer feature injection (early encoder layers → early LLM layers) preserves low-level acoustics (prosody, timbre) alongside semantics, so one model answers both “what did she say” and “does she sound stressed”. Runtime thinking-budget control on open models: a custom logit processor + request param (thinking_budget: 0 | N | unlimited; 0 skips thinking; N injects the official transition sentence rather than force-closing the think tag, avoiding tag duplication) plus flags to split reasoning_content from content in streams — one reasoning-cost knob mappable to both hosted thinking budgets and local logit processors. Instruct-vs-Thinking post-trainings of the same base are a product axis: direct-answer variants win transcription/captioning; CoT+RL variants win reasoning benchmarks — task type dictates variant, not size. Transcripts should persist timestamp anchors as retrieval keys so answers can cite seekable positions (see knowledge-pipelines).

17. OpenAI-compatible serving as the universal interop seam

Expose local capability behind the de-facto wire protocols: /v1/audio/speech (model alias tts-1 → active engine, or concrete engine id; voice maps standard names via an alias table, any other value resolved as a voice-profile id), /v1/audio/transcriptions (json/text/verbose_json/srt/vtt; verbose_json can carry diarized speaker segments), /v1/audio/voices, /v1/chat/completions with audio content parts, and Responses-API-compatible agent endpoints. Two hard-won details: declare all extension params explicitly in the request schema or the framework silently discards them, and use separate timeout budgets mapped to distinct status codes (model-load timeout → 503 “warming”, generate hang → 500 “stuck”, ASR → 504). The inverse also holds: remote-offload is just-another-backend — a pure network-client engine (server URL + model + encrypted key, “Test connection” hits GET /models and classifies failure) satisfying the same provider contract, with the rule that auto-detect never selects a remote engine. See ecosystem-and-interop.

18. Browser-side inference lane

ONNX Runtime Web on WASM gives app UIs zero-backend inference for small models (18–243 MB): model binaries streamed from a hub, cached as raw ArrayBuffers in IndexedDB keyed by the catalog’s versioned cacheKey, sessions created lazily and memoized in a ref map, numThreads = 1 to avoid SharedArrayBuffer/COOP-COEP header requirements. Split timeouts: ~90 s session creation vs ~120 s inference via a generic Promise.race helper. Pipelines are shape-driven (letterbox into fixed input, planar CHW float32, graph input name from catalog data, map outputs back through the letterbox transform onto original resolution), with defensive output normalization (conditional sigmoid when values fall outside [0,1] so both logit- and probability-output models work). Hold large results as Blobs, not base64. Quant vs FP16 variants as a user-facing size/quality axis, defaulting to quantized; size-parsed tier icons (<40 MB light / <60 MB standard / else heavy).

19. Kernel-style scheduling and preemption for shared local inference

Treat agent-resource interaction as an OS problem: every request is a typed work unit (“syscall”) with agent identity, status lifecycle (pending/executing/done), start/end timing, and a completion event; per-resource-type queues and scheduler threads (LLM / memory / storage / tool) prevent head-of-line blocking across heterogeneous work. Round-robin time-slicing can preempt local HF-transformers generation mid-stream by checkpointing KV cache + generated tokens + position, then resuming — true preemptive multitasking of one GPU across agents. Cheaper cooperative variants for most systems: the interactive gate + Semaphore(1) of mechanism 9.

20. Local-vs-cloud routing and efficiency telemetry

Layered routing, convergent across systems: (a) heuristic tier — static rules (code → coder model, math/long → largest, short → smallest, urgency > 0.8 → smallest); (b) trace-driven tier — classify queries into ~5 classes, score models per class as 0.60 × success_rate + 0.40 × avg_feedback, require ≥5 samples before trusting a learned preference, conservative online updates; (c) cost-optimal tier — ILP over live token pricing + historical performance. Reward shaping example: latency 0.4 + cost 0.3 + token-efficiency 0.3 (normalized); training-composite example: accuracy − z-scored energy/latency/cost penalties with weights 0.5/0.1/0.1/0.3. All of it depends on per-step telemetry: tokens, TTFT, latency, energy joules, power watts, GPU utilization, throughput recorded on every inference step in the run ledger, with tool steps tagged by skill so optimizers can bucket. Teacher/student escalation (weak local model runs; detected failure escalates to a frontier model that fixes and distills) is the routing pattern’s learning arm — mechanics in self-improvement-loops. Model-name-derived context budgeting is a crude-but-useful default: parse (\d+)b from the model name, scale (7B → 4096 tokens, ×1.5 per size step, round to power of two) to drive compression triggers; better: store ideal_ctx in the catalog.

21. Weak-model harness adaptations

Designs that make small local models reliable: fenced-code-block tool protocols (extract ```tag blocks from raw text; first line optionally names a save path; execution feedback pushed back as a user message; executed blocks replaced by block:N placeholders re-expanded by the UI) — works on any model that can emit markdown, no function-calling API assumed; multi-dialect text-parsing fallbacks (Action/Action-Input, <tool_call> XML, inline attribute tags) for non-native-tool models; plain-text sentinel vocabularies (NO_UPDATE, REQUEST_CLARIFICATION, GO_BACK, REQUEST_EXIT) instead of structured actions; attributed context injection (“According to step X: …” beats bare concatenation on weak models). Tiny local classifiers as zero-token routers: zero-shot NLI + a prototype-memory adaptive classifier (prototype weight 0.8 / neural head 0.2, EWC lambda 100 for continual learning from corrections) with relative-confidence voting, ~150 few-shot seeds, and the key inversion — low classifier confidence escalates to the expensive path, classifier exceptions fall back to the cheap one. A small local seq2seq summarizer handles context compression without cloud calls. See agent-harness-engineering for the harness-side composition.

22. Release manifests and idempotent installers for native binaries

A pinned-release JSON manifest for engine binaries: release tag + per-OS/per-accelerator archive filename templates with a {release} placeholder + env override. Installers are idempotent-repair: existence-check every component, .part + rename downloads, prebuilt binaries verified by executing --help, fall back to source compile, versions pinned everywhere. Optional third-party advisors run as subprocesses with a JSON contract, cached by a hardware hash (cores-cpuModel-gpuName-ram, ~1 h TTL), always with an internal fallback — pluggable intelligence that can never become a hard dependency.

23. Process supervision lifecycles

Resident engines (LLM server, diffusion server) vs per-request spawns (CLI ASR with output flags, TTS as a worker process speaking JSON over stdin/stdout with in-process float32→PCM16 conversion) are both valid; choose by load cost. Graceful shutdown discipline: SIGTERM → 2.5 s wait → SIGKILL → up to 5 s wait for port release. Port management: preferred port + a per-service scan range, tested by binding a throwaway server before use. Inject native lib dirs into PATH/LD_LIBRARY_PATH for spawned CLIs rather than requiring system installs. On host boot, sweep zombie state: mark stale “running” records aborted (“server restarted”) and push overdue scheduled work +60 s so a restart doesn’t fire everything at once.

24. Staged progress from log-pattern matching

Engines with no structured progress reporting still get honest coarse progress: map known log substrings to staged percentages (“Loading unet” → 62%), clamp to 99 until truly ready, and feed one uniform progress shape {active, phase, progress, current, total, speed?, detail} that the UI renders identically for exact (tensor-level) and inferred (stage-level) progress. Serve-side crash watchdogs poll the supervised process at escalating intervals (25 s/60 s/2 m/5 m) for exit markers, deregister the endpoint on non-zero exit, and regex-match the last ~6000 chars of output against ~a dozen known failure signatures (CUDA OOM, tensor-parallel divisibility, gated repo…) returning structured retry suggestions (--gpu-memory-utilization 0.95, --max-model-len 2048).

Patterns & compositions

• Catalog → download → probe → benchmark → persist → serve → watchdog. The full model-ops pipeline composes mechanisms 3, 4, 7, 22, 23: declarative catalog entry, resumable verified download, probe-by-execution, per-backend benchmark with persisted winner, supervised serving with failure diagnosis. Each stage emits machine-readable state a UI can reattach to.
• Capability-driven node expansion. Workflow/pipeline steps bind to a capability requirement set ({transcribe, speaker_labels}), and the resolver picks ONE provider satisfying the set (joint model) or compiles a sub-graph of providers (STT ∥ diarizer → fusion). Templates stay stable while the model inventory changes underneath. Requires the structured capability matrix (mechanisms 12, 15).
• Remote as just-another-provider. The same engine ABC accommodates a pure network client (OpenAI-compat URL + key); serving your own stack behind the same wire protocol closes the loop — any box running the open serving stacks satisfies the local contract remotely. Auto-detection only ever picks local.
• One reasoning-cost knob, two backends. A single reasoning_budget request parameter maps to hosted thinking budgets on API models and to logit-processor budget injection on local models (mechanism 16) — callers never care which.
• Measure-then-remember. Benchmark winners, trace-driven routing scores, and fit verdicts all persist as {value, evidence: {metric, measured_at}} records with provenance, so preferences are explainable and stale measurements can decay. Multiple independent systems converge on this shape.
• Everything yields to the human. Interactive gate + cancel-and-defer + queued-row-before-semaphore compose into “background means background” on shared local hardware; UIs distinguish queued / running / deferred / skipped truthfully.
• Trait-tagged tiered search. Server-side classification of hub search results (params-B parsed from name, vision/uncensored/code traits via regex), ranked by downloads + tier-match bonus + query-match bonus, short-TTL cached (5 min) keyed by tier+query, sizes fetched lazily and combined with fit reasons — search returns one right file per repo with an honest “why it fits”.
• Capability matrix as an org-level product shape. Mature model families ship a matrix, not one model: a 0.9B pipeline workhorse (joint transcribe+diarize), a 2B single-task ASR, 4B/8B understanding models in Instruct/Thinking variants, a tokenizer/feature extractor, and a separate TTS line — size scales with task breadth. A local-model platform should mirror this: bind small specialists to high-volume use-cases and reserve large generalists for open-ended reasoning, per the declared capability matrix rather than a single “best model” setting.
• Post-turn maintenance serialization. Background LLM work triggered by inference (memory extraction, indexing, self-observation) runs through an explicit post-answer queue so maintenance never competes with response generation for the local model — a clean architectural boundary multiple assistant-shaped systems adopt.

Anti-patterns & failure modes

• is_available() -> bool. Loses the why; every “engine missing” becomes an unactionable dead toggle. The message-carrying tuple is strictly better and costs nothing.
• Probing one path for download/delete detection. Hub caches, save layouts, and sidecar dirs diverge; checking only one layout showed every model “not downloaded” while inference ran live, and a delete that missed a layout left phantom bindings. Enumerate every path each code path writes.
• File-format ≠ runnable. A GGUF quant existing does not mean the standard runtime supports the architecture (custom feature-injection layers need custom runtimes). Feasibility checks must test the runtime, not the file extension.
• Fit gates in the binding path. Fit estimation drifts with quant/context/runtime changes; hard-blocking bindings on it produces false “won’t run” states. Fit belongs in search/recommendation with reason strings; binding should try, then degrade via the OOM ladder.
• Reporting 100% before persistence. Progress landing on 100 before the artifact is durably saved produces “downloaded” models that aren’t. Cap at 99 until the post-persist ack.
• One timeout number. A 10-minute download is fine; 30 s of silence is not. Systems without a stall timeout distinct from total timeout either kill healthy long operations or hang forever on dead ones.
• In-process native libs. Encoder/loky/torch teardown segfaults kill the host process and strand stores mid-migration. Sidecar isolation is structural; thread-pinning env vars only narrow the window. Similarly: destructive model tests without a models-dir sandbox have deleted users’ real bound models.
• Prompt-window hotword overflow. Piling a lexicon into an ASR bias prompt past the model’s window fails silently with empty transcripts — no error. Per-provider hotword_budget caps (one family: ~200 chars) must be enforced at the wiring layer.
• Conflating provider JSON-mode with constrained decoding. Fine-tuned “JSON mode” has no guarantees; published negative results on structured output traced to this conflation plus schema-absent prompts and mismatched prompt pairs. When fairly compared, structure wins or ties everywhere.
• Constraints without reasoning room. Tightly constrained answers with no bounded reasoning field before them measurably suppress chain-of-thought and accuracy.
• Preset UI before mechanism. Quality presets rendered in a picker while the execution path hardcodes one value — dead wiring that erodes trust. Audit manifest/UI claims against the actual call path.
• GPU telemetry during heavy inference. Driver-query polling while a diffusion/LLM backend runs contends with inference and lags the display; pause polling while heavy backends run, throttle ≥4.5 s otherwise.
• Reattach ≠ resume; no integrity checks. A UI reattaching to a running download is not byte-range resume; and downloads without hash/size verification admit truncated weights. size_gb expected-size checks are the minimum viable detector.
• Single global download slot serializes all model acquisition behind one transfer; per-kind slots or a queue are cheap.
• Health = 200 OK. Liveness without identity verification (which model does this port serve? is this the child I spawned?) admits stale-server and stale-generation bugs.

Quantitative findings

• Local models handle 88.7% of single-turn chat/reasoning queries (intelligence-per-watt research thesis).
• Constrained generation on GSM8K, 8 models: structured always ≥ unstructured, up to >70% lift; prompt-format sensitivity collapsed from 63.4%→18.9% (unconstrained) to 73.8%/71.1% (constrained). Fair reruns of the “structure hurts” claim: 0.78 vs 0.77, 0.77 vs 0.73, 0.44 vs 0.41 (structured vs not). Flexible-regex answer extraction beat an LLM-as-parser 0.61 vs 0.57. CoT length bounds of min 50 / max 700 chars shaped reasoning; tighter caps (300/500) shrank gains.
• Joint 0.9B transcribe+diarize beats GPT-4o/Gemini-class and commercial diarization on meeting/podcast cpCER; bf16 ≈ 2 GB, CPU fallback. A 2B English ASR sibling: 4.87% avg WER (open leaderboard). Time-marker models: timestamp-ASR AAS 35.77 / 131.61 vs 833.66 / 708.24 for 30B-class omni models — ~6–20× temporal-grounding gap. Single-H100 serving: ~7 req/s short clips, 98.8 audio-seconds/sec at concurrency 16.
• Fit math: runtimeNeed = size × 1.25 + 1.5 GB; reserve max(4 GB, 30% RAM); verdict ratios ≤0.50 perfect / ≤0.78 good; discrete headroom ≥1.5× perfect; raw_tps = bandwidth / model_GB × 0.55, CPU-offload harmonic blend at cpu_bw 55 GB/s; KV-cache VRAM multipliers q4 0.25× / q8 0.5× / f16 1×; 85% VRAM headroom; +0.8 GB load safety margin.
• Tier thresholds: unified RAM ≥16 GB high / ≥8 mid; VRAM ≥12 GB high / ≥6 mid. Quant ladders: high q6_k→q5_k_m→q4_k_m→q8_0; low/mid q4_k_m→q4_k_s→q4_0→q3_k_m→iq4_xs. OOM context ladder: 32768…512 (9 steps, 1 retry). Community hardware guidance: 7B hallucination-prone for agent work, 14B simple tasks, 32B/24 GB VRAM most tasks, 70B+ advanced.
• Timeout constellation that works: 90 s session/model create, 120 s inference, 10 min download total, 30 s stall, 90 s bounded self-test; SSE comment heartbeats every 10 s; replay buffers evicted 180 s after last subscriber.
• Interactive gate: 1.5 s HTTP quiet window, 45 s browser-heartbeat staleness, 15 min defer on user return (escalating ≥40 min after 2 defers).
• Trace-driven routing: per-class model score = 0.60 success + 0.40 feedback, ≥5 samples before trust; reward = latency 0.4 + cost 0.3 + token-efficiency 0.3; learning-composite weights 0.5/0.1/0.1/0.3 (accuracy/energy/latency/cost).
• Voice: zero-shot cloning from a 3 s clip (646 languages claimed); 3–10 s reference at ≥ −15 dB is the quality sweet spot; echo filter threshold = 3 consecutive shared words; one ASR family’s bias-prompt budget ≈ 224 tokens (~200 chars safe cap).
• Browser ONNX: useful models at 18–243 MB (bg-removal quantized ~25–44 MB); tier badges <40 MB / <60 MB; benchmark ring buffer cap 100; hub search cache TTL 5 min; hardware-hash advisor cache 1 h.
• Local footprints on the audio matrix: 0.9B joint transcribe+diarize ≈ 2 GB bf16 (CPU-viable); 2B ASR ≈ 4.8 GB; 4B understanding ≈ 9.7 GB f16 / 2.75 GB Q4_K (custom runtime); 8B ≈ 17 GB bf16 — high-RAM machines only.
• Router/classifier configs that ship: two-classifier relative-confidence voting; ≤8-char short-circuit to “chat”; complexity gate defaults HIGH below 0.5 confidence; adaptive classifier prototype weight 0.8 / neural head 0.2, EWC λ=100, prototype update every 50 examples.

Open questions

• Streaming ASR from unified audio-LMs: every verified inference path for joint/understanding audio models is offline/batch; max audio duration and VRAM minimums are unpublished. Real-time local duplex still requires the classic streaming stack (Kaldi/whisper-class) — when do audio-LMs get streaming decoders?
• KV-cache checkpoint preemption in practice: demonstrated for HF-transformers generation; unproven for llama.cpp/MLX/vLLM-class runtimes where most local serving actually happens. Is the cooperative gate + single semaphore the practical ceiling?
• Where does fit belong long-term? The corpus splits: some systems bake fit into ranking and serving defaults; others deliberately removed fit flags from binding after false negatives. Search-only fit with honest reasons is the current consensus, but nobody has validated fit-estimation accuracy against measured loads at scale.
• Energy telemetry portability: per-step joules/watts is proven valuable, but the measurement story outside instrumented research rigs (e.g. macOS powermetrics sampling attribution to a single process) is unresolved.
• Constrained decoding across the API boundary: provider structured-output capabilities drift (schema subset support, silent param discard); no system yet maintains a live-probed capability matrix rather than a hardcoded one.
• Browser-lane vs gateway-lane split: browser-side ONNX inference offloads the host and sandboxes naturally, but fragments the catalog/cache/telemetry story into two stores. No system in the corpus unifies them.
• Model-boundary content scanning under streaming: wrap-the-engine guardrail scanners (secrets/PII, WARN/REDACT/BLOCK) admit they cannot BLOCK already-streamed tokens — input-side only for streams. An output-side design compatible with token streaming is open; see security-and-guardrails.