Workflow Engine Design

Part of the PersonalClaw research-learnings library. Source-agnostic; distilled 2026-07-13 from a 95-source competitive-research corpus.

Scope: graph/DAG execution engines — node taxonomies, declarative specs, scheduling & frontier computation, joins & conditional convergence, retries & failure envelopes, timeouts, effect idempotency & side-effect ledgers, rewind/fork/replay, journaling, mid-flight mutation, budgets, structured stage outputs. Sibling topics: multi-agent-orchestration, automation-and-triggers, verification-and-judging, planning-and-decomposition, agent-harness-engineering, self-improvement-loops, security-and-guardrails.

Principles

Control-flow ownership is a design axis, decided per task, not per product. The canonical three-tier taxonomy: task = one model call (bounded failure modes, predictable cost); workflow = multiple calls in predefined control flow (“the control flow is yours”); agent = a model using tools in a loop, deciding its own trajectory (“the control flow belongs to the model”). A 4-question checklist routes between tiers: (1) can you pre-map the decision tree? → build it explicitly as a workflow and optimize each node; (2) does value justify token spend? (~10¢/task ≈ 30–50K tokens is workflow territory); (3) is the bottleneck capability reliable? errors compound per loop iteration; (4) how costly and how discoverable are errors? high-stakes + hard-to-detect makes autonomy a liability. “Labels matter because architecture follows them.”
The counter-thesis is real and must be designed against. A 27k-star production framework deleted its explicit pipeline/orchestration layer in a major rewrite, replacing declarative graphs with tool-driven coordination and agent-managed task lists, betting that increasingly agentic LLMs need less structure. The synthesis: a graph engine survives only if the agentic escape hatch is first-class — cheap NL→spec generation and cheap typed mid-flight mutation — or the graph becomes rigidity users route around.
Structure buys variance reduction, not just correctness. Measured evaluation methodology across paired workflow-vs-single-prompt runs shows the largest lift on RP@k (share of tasks where every run of k passes) — workflows’ value is reliability. The thesis, stated explicitly in one corpus source: the win is process discipline, not model capability, and checkpoints also absorb silent base-model quality drift.
Event-sourced state law. Define the journal/UI contract as: folding a run’s event stream reconstructs run state exactly (one message accumulates from its events via a pure append_event fold); ship the same fold in FE code; SSE replays buffered events to late joiners so reconnects and multiple tabs converge byte-equal. Run/plan status is always a derived projection of node states (sticky cancelled; running if any child runs; failed-if-any-failure-else-completed when no pending remain) — never a directly writable field.
Termination and payload are orthogonal concepts. “Are we done?” (a completion signal: pure marker string, engine never injects it into the prompt) and “what did the node produce?” (structured output: tag + schema validation) are separate node contracts; a run can use either, both, or neither. Conflating them corrupts both.
Schema without semantics is dead weight. One architecturally serious system shipped checkpoint_id, retry_count, state_snapshot, and precomputed parallel levels — and its executor was sequential, fail-fast, resume-free. Engine acceptance criteria must exercise resume/rewind/parallelism/retry behaviorally, or the fields rot as false documentation.
Correctness first; everything else is downstream optimization. An agent node reduces to environment + tools + system prompt; caching, parallel tool dispatch, and progress surfacing come only after the basics are stable, because upfront complexity kills iteration speed. Multiple independent systems converge on this ordering.
Fail fast on prompt/context assembly; let the orchestrator own retries. A retried or degraded dynamic prompt expansion feeds the model a prompt assembled in a broken environment — worse than a clean abort in AFK contexts. Retry is safe for idempotent infrastructure, unsafe for prompt content. Node-internal assembly failures return typed diagnostics (elapsedMs, exitCode); the scheduler layer that owns parallelism makes the retry decision.

Mechanisms

Node taxonomy

A converged working set of node kinds (assembled across several independent engines):

infer / single-call — one bounded LLM completion, no tools, no agent spawn; for classify/extract/score/synthesize steps. The cheapest tier; most engines lack it and overpay with full agent sessions.
classify-then-route — an enum-constrained LLM call returning exactly one of output_choices, with a route: map from choice → downstream node/subgraph. Makes branching cheap, auditable, and routable to cheap models. Paired with a branch/switch node (on: binding, cases: {label: node}, default).
agent/stage — full tool-loop session; the expensive tier.
tool_call — deterministic execution with a mandatory tool allowlist.
user_input / clarify wait — pause with a typed form: field kinds string/enum/int/bool, defaults, cancel keywords, 24–48h timeout, and auto-suppression on unattended/cron runs. Renders as a real form in a needs-input inbox, not a chat prompt.
wait — mode: time (1ms–1y, ~5s poller resolution) or mode: event: pause until a bus event whose payload passes a filter; scope: run|global; optional timeout routed via a dedicated timeout port. Filters: object form = dot-path deep-equal; string form = a sandboxed predicate with shadowed globals, cloned payload, 50ms hard wall-clock cap, 2KB source cap.
gate / HITL — typed question forms (approval/text/single-select/multi-select/boolean), approver policy any|all|{min:N}, timeout with action: reject, resolution resumes via approved|rejected|timeout ports; idempotent per step id.
sink — a configured terminal action provider (log/persist/notify/REST) with a declared input schema; saved instances fold input_defaults.* into fixed inputs, leaving the rest as wirable ports.
trigger nodes are entry-only and excluded from “did anything complete” checks (see failure envelopes). Two spec-level modifiers: final_output: node:<id> | auto | raw pins which node’s output is the user-facing answer (letting an audit node be the workflow’s mouth), and per-node-instance flags disabled (mute: skip but keep in spec), continue_on_fail, retry: {max_tries}, type_version (pin node semantics for migration).

Declarative spec & compilation

Nodes-with-next, edges derived: next: string | string[] | Record<port,string> — fan-out is an array, conditional routing is a port map; executors return an optional nextPort. Alternative: explicit port-to-port edge lists {source_node, target_node, source_port, target_port} with typed connection channels. A third proven carrier: a markdown skill file whose frontmatter contains a composition: DAG — one file format serving both atomic capability and workflow.
Blueprint → compiled immutable spec: the editable definition compiles into a versioned, immutable spec (resolved_config, graph_snapshot, node_specs, compiler_version, validation_errors, unique (id, version)); constants fold at compile time; every run pins to a spec version and agent nodes pin to definition-version snapshots — protecting in-flight rewind/fork from concurrent spec edits.
Mechanical parameter derivation: resolve_unfilled_inputs() — inputs neither wired nor statically bound automatically become the launch-form/parameter schema. No hand-maintained parameter lists; forms can’t drift from the spec.
Kahn validation at create time: cycle detection is a create-time failure; a topological pass assigns each node a persisted execution_level (same level ⇒ safe to run concurrently), and validation returns the level-grouped execution order so reviewers/UI see concurrency structure before anything runs.
Type-compat as warnings: port type mismatch emits warnings, not errors, with “text” as a universal source type.
Deployment = instance-of-definition: a spec is inert until deployed with baked inputs + a trigger binding + health rollups (last_run_at, last_success_at, last_failure_at) and optionally an owned workspace (workspace_provisioning: none|auto) for cross-run continuity. See automation-and-triggers.

Frontier computation & joins

Kahn-style in-degree frontier: track remaining in-degree per node (-1 = launched); launch all ready nodes in parallel; non-blocking with node_started/node_completed/execution_finished signals; soft-cancel flag.
Active-edge convergence gating (the critical join subtlety): the walker tracks edges for actually-taken paths only; a convergence node waits only on predecessors with an active edge into it — so a conditional branch that never ran can’t deadlock the join. Second subtlety: waiting/async steps must register their structural outgoing edges as active at wait-entry, or a 3-way fan-out with 1 fast + 2 async branches fires the merge after the first completion. Two regression tests fall out: (a) untaken branch must not deadlock the join; (b) 1-of-N async completion must not fire the join early.
execute_from(start_node) — re-run only the subgraph downstream of a node with upstream outputs pre-satisfied from the journal: the third re-entry primitive besides resume and rewind, cheap on a frontier scheduler (prune to descendants, seed in-degrees).
Pull-based frontier as an external protocol: create → get ready set (+ resolved params) → external actor executes → report result → next ready set. The plan store becomes a pure state machine any orchestrator (LLM, human, remote client) can drive; the engine only validates transitions (reject terminal-task updates, refuse running/completed if deps unsatisfied) and journals.
Defensive router path maps: every conditional edge maps the complete set of router return values so a fall-through label (prompt/i18n/refactor drift) can never hit a missing entry and crash mid-run.

Loops, iteration identity, and circuit breakers

Iteration-aware step identity: step key = (run_id, node_id, iteration) where iteration = count of existing steps for that node. Memoization/dedup applies within an iteration; loop targets re-execute in new iterations; context reconstruction injects the latest step per node. Keep two dedup sets: per-walk (blocks double-execution within one scheduling pass) vs per-run (allows a prior walk’s loop target to re-run).
Two-level circuit breakers: max node executions per walk (~100, “possible infinite loop”) + max total steps per run (~500) — the run-level cap checked at every walk entry so async resumes and retry-poller re-entries are covered. Breaker trips are terminal states with reason strings.
Loop-node health autopilot: auto-pause after N consecutive failed iterations (default 3); auto-pause when evaluation output is byte-identical (canonical JSON) across M cycles (default 4 — “stuck”); warn when cycle duration exceeds 5× rolling average. All computable from journal data, zero LLM cost.
Agent-proposed wake time: a long-running cyclic node’s structured output may include next_cycle_delay_seconds (default 300), clamped by configured cadence — adaptive polling.
Loop exit predicates worth first-class support: consecutive_clean: N (exit only after N consecutive independent clean judge passes — the “double-clean” rule, empirically a stronger convergence criterion than single-pass approval) and rubric-target completion (see verification-and-judging).

Failure envelopes

Multiple independent systems converge on a three-policy per-node failure vocabulary, plan-default + per-node override:

stop — abort remaining work; skip_dependents — BFS over reverse edges, skip only transitive dependents; continue — downstream convergence receives a [FAILED: reason] sentinel and handles partial results. Readiness rule: a failed dependency satisfies a successor iff its effective policy is continue.
Partial-failure terminal semantics: distinguish all-branches-failed (run failed) from partial failure (run completed + error string naming failed nodes), excluding entry/trigger nodes from the “did anything complete” test — otherwise a linear trigger→validate→action run that fails validation reads as “partial success”.
Cascade-fail dependents: a failed/cancelled/superseded parent flips blocked dependents to a terminal state with a descriptive reason — blocked-forever children are a bug class.
Per-node on_failure substitute step, with two sanity constraints: the substitute has no dependencies and no nested on_failure.
Retry policy per step: exponential-with-full-jitter / linear / static with a maxDelay clamp, executed by a background poller off a persisted nextRetryAt column (survives restarts); a validate step can pass/halt/retry, and each retry appends to a ${node}_validations history array — the retried node sees all prior verdicts, never an overwrite.
7-state terminal outcome taxonomy: succeeded | failed | crashed | timed_out | no_change | scope_violation | discarded — crash ≠ timeout ≠ produced-nothing ≠ wrote-outside-scope. no_change inherits the parent’s prior evaluation instead of re-running downstream scoring — a real scheduler optimization.
Failure events carry a taxonomy: failure_class, error_code, severity, retryability (retryable/not_retryable), affected_node_key, resolved flag — the classification is what turns failures into retry policy and lessons (self-improvement-loops).
Attempt budgets: attempts = max(max_attempts, repair_rounds+1); a repair attempt appends failure stage/summary + prior summary (~600 chars) + “make the smallest patch”, and does not reset a branch that has valid progress.

Timeouts

The dual-timeout model (the standout mechanism for agent nodes): idle timeout before any completion signal (default 600s, reset on every output event) — genuinely stuck → fail; completion timeout after the signal is seen but the process won’t exit (a spawned child holds stdout open) — a ~60s silence-based grace window, reset by each output line so trailing token-usage/structured-output events are captured, and on expiry the run resolves successfully with a warning, work preserved. Without the split, a done-but-not-exited agent waits out the idle timeout and fails, discarding committed work.
Per-node timeoutMs via a race against the executor promise (default ~30s for deterministic nodes), plus a separate wall-clock budget for any spawned script — orchestration timeout ≠ payload timeout.
Mode-dependent gate timeouts: approval waits get ~30s under scheduled/unattended fires (fail fast, observably, with a distinct timed_out_unattended event) vs ~600s attended. Stronger variant: unattended policy converts ask→deny wholesale, keeping tool safety checks live while never parking (security-and-guardrails).

Structured stage outputs

Tag + schema extraction: a typed payload extracted from an XML tag in agent output; schema = any standard validator. Rules proven in production: the caller owns the prompt-side instruction and the engine fails at compile time if the resolved prompt lacks the opening tag; last match wins (agent self-correction is frequent and benign — also protects against grabbing quoted/example blocks); fence-aware (unwrap ```json) but otherwise strict JSON — tolerant parsing hides genuine model failures; failure throws a typed error carrying tag, raw match, cause, and all preserved side-effect state (commits, branch, session id) so the caller decides recovery.
Retry = resume the same session with token-efficient error feedback, not redo: maxRetries re-enters the failed session and feeds the validation error back so the agent re-emits a corrected tag without redoing the work. “Resuming the session IS recovery.”
Produce-then-extract two-phase pattern: run the expensive work without output constraints, then a 1-iteration extraction run that resumes the work session with a dedicated extraction prompt + schema + retries — keeps schema-emission pressure out of the work prompt and makes extraction independently retryable.
Alternative conventions with the same last-match discipline: a fenced ```output block parsed last-match with declared-keys-only merge (raw text always kept under a fallback key; strip \x00 before persisting to JSONB); a last-line JSON contract for subagents (prose reasoning first, ONE final JSON line; parse failure → log full stdout, mark INVALID, re-prompt once or surface — never silent-fail).
Completion-signal contract: default marker string (e.g. <promise>COMPLETE</promise>), string or array, first match stops the loop, pure termination marker with no payload, engine never injects it into the prompt.

Bindings & data flow

Default-deny cross-node access: only trigger/input/workflow-scope names are in interpolation scope; reading another node’s output requires an explicit inputs: {localName: "node-id"} mapping. The documented footgun to reject: unmapped references silently resolve to empty strings — make unresolved references a hard node failure with journaled unresolvedTokens diagnostics.
Type preservation, two-regex rule: an exact-single-token value ("{{node.output}}") preserves the resolved native JSON type (dict/list/int); a token embedded in text stringifies (json.dumps for containers). This distinction must be explicit in the binding language.
The floor to avoid: a whole-value-only expression engine without filters or inline interpolation immediately spawns ad-hoc regex parsing inside nodes (a conditional node regex-parsing comparisons out of strings). Ship filters and inline interpolation from day one.
Sanitization as lintable law: untrusted-origin bindings (user input, upstream free text, trigger payloads) flowing into prompt positions must pass filters (xml_escape | truncate(n) | tojson | slugify); an unfiltered untrusted binding is a spec-validation error. Two injection rules from a second system: interpolation/expansion applies only to template-sourced prompts (programmatically built strings are literal), and expansion markers inside substituted argument values are inert text.
Secrets: {{secret:KEY}} resolved server-side at execution; specs, journals, audit logs store the template, never the value. Complementary: redact-at-journal-write, live-at-execute — resolved secret inputs are scrubbed from persisted step records (a taint set computed at input-resolution time) while executors see real values, so run-inspection APIs can never leak credentials.

Journaling & durability

Checkpoint per step: after every step an atomic transaction writes step result + merged run context. On recovery, handle four cases: interrupted running runs (recompute ready nodes from completed steps + active edges), waiting runs whose async work finished while down, unresolved approvals, overdue waits. Re-validate recovered outputs against the executor’s output schema before reinjection — corrupted checkpoints are skipped and journaled, not trusted.
Storage discipline: per-run/per-definition files so history reads are O(history), not O(all runs); atomic writes via mkstemp + fsync + os.replace (power-off mid-write must not truncate; loaders that skip unparseable JSON make corruption silent — design against it); trim caps (e.g. 200 runs/definition); an append-only JSONL diff-audit per definition ({ts, who, diff: {key: {before, after}}}) with a 256KiB soft cap truncating to the last ~128KiB.
Terminal-write ownership: only the frontier executor writes a run’s terminal status; cancel/pause endpoints write signals the loop consumes (otherwise an HTTP handler overwrites a stopped-by-user failure with success). Run-side persistence re-reads the record and mutates only its own field family, with += deltas not assignment, so engine writes and user PATCHes can’t clobber each other; deleted definitions are never resurrected by a run save.
Journal the resolved prompt (post-binding text or content ref) per stage instance, plus per-node LLM telemetry: latency, input/output/reasoning tokens, cost, actual model+endpoint served — enables trajectory replay (“what did the model see”) and gives template evaluation cost ground truth.
Event-stream contract: broadcast the step advance before dispatching the node (UI flips immediately); a cheap watcher surfaces the latest tool-call label; all event publishes wrapped so a broken subscriber can never kill the run; on WS reconnect the gateway replays announce → state → completed events.
Optimistic concurrency without locks: SHA-256 of canonicalized state (revision stripped) stored as _revision; writers re-check on-disk revision before tempfile+rename; conflict → typed error + retry hint. Also If-Match on updated_at → 409 stale_update for spec PATCHes.
Dangling-tool-result invariant: any journal cut (rewind epoch boundary, context truncation) must walk back until every retained tool result has its originating tool call — naive cuts produce API-rejected conversations on resume. Interrupting a parked run must synthesize interrupted tool-results for every pending call (start/delta/end) before ending the reply, so the journal never contains half-open calls; interrupt of an idle run is a recorded no-op.

Effect idempotency & the side-effect ledger

Effect ledger: every side-effecting dispatch records {idempotency_key: sha256(run_id + instance_path + epoch), effect_status: attempted|committed|retried|compensated|skipped, compensation_ref?} in the journal. The output cache memoizes values; without effect identity, resume/rewind/fork double-fire external effects (message sends, entity creates). Rules: rewind/fork across a committed effect surfaces it in the cascade preview instead of silently re-running; re-executing a node with a committed effect requires explicit redo_effects: true; run-start and mutation APIs accept a caller idempotency key with a short-lived dedupe cache (a retried tool call returns the existing run id).
Idempotent state classification before any mutation (for trigger-fired and re-entrant runs): classify FRESH (run from step 1) / ALREADY_DONE (report success, exit) / PARTIAL_DRIFT (resume from first unfinished step after verifying completed steps’ outputs) / UNEXPECTED_DRIFT (markers contradict — stop and escalate, never force forward). A re-fired trigger becomes a no-op instead of a double-apply.
Re-runnable hooks are a documented contract: environment/setup hooks run on every resume and must be idempotent (guard each step, marker files for one-time work); nodes declare idempotency or a checkpoint marker. Idempotent timing: COALESCE(started_at, now) so the first timestamp wins under retries.
All side effects of a deduplicated write share one idempotency key: a measured bug — content section deduped but a salience counter still incremented on re-ingest, inflating forever. Key every mutation (body, link records, counters, index entries) on (source_id, target, op) so engine replays are exactly-once.
Provisioning safety: external-resource providers follow a contract of stdout = one JSON object, stderr = journaled logs, and a paired teardown command receiving the output id, with teardown required idempotent — makes rewind/fork over a provisioning region safe.

Rewind, fork, replay

Three re-entry primitives: resume (continue from checkpoint), rewind (re-execute from an earlier epoch), and run-from-node (downstream subgraph re-run, upstream pre-satisfied). Plus replay --dry-run: re-materialize the frontier plan from the journal without executing.
Split rewind into rollback vs revert: rollback = hard reset to an epoch with preserved forward references; revert = inverse-patch one node’s effects, refusing with a conflict-naming 409 when later state overlaps. Both require a mutation preview before destructive apply.
Archive-before-overwrite: in-place re-execution snapshots the superseded journal region + produced artifacts (<artifact>.backup.<ISO-ts> + a change-history log) — “even confirmed re-execution occasionally needs the prior state.”
Fork must declare which axes it isolates. A session-only fork (native fork flag; parent transcript byte-for-byte unchanged) does NOT isolate branch/workspace/sandbox — safe concurrent fan-out requires per-fork disambiguation (distinct branch/dir) or the engine forces it. Corollary: unique-name generators need randomness — second-granularity timestamps collide for forks fired in one tick.
Shape-signature checkpoint invalidation: checkpoint thread id = sha256(subject : date : graph_shape_signature)[:16] where the signature folds every shape-affecting config — a resume under a mutated graph hashes to a different thread and starts fresh instead of silently reusing incompatible state. Generalization: stamp journal epochs with a hash of shape-affecting spec regions; resume/fork under a changed spec forces re-frontier.
Resume invariants: budgets are pre-charged with journaled consumption (“resume must not mint a fresh budget”); HITL answers are consumed atomically (one-shot rows deleted in the same transaction that resumes) so double-resume can’t replay them; wake-type resumes are droppable if the run is already active, gate-answer resumes must re-queue until the parked run takes them.
Session capture posture: transcripts captured after each iteration with cwd fields rewritten so native resume works on the target host; auxiliary transcripts best-effort, but main-session capture failure fails the run — a run you cannot resume or audit is a failed run.

Mid-flight mutation

Draft/commit staging: pending edits accumulate in a draft_steps shadow; scheduled/running executions only ever read committed spec (the frozen-region invariant in its simplest form); explicit commit promotes, discard drops; while a draft exists, spec-touching PATCHes stage silently into the draft.
A change-case taxonomy with cascade computation (8 cases): add-skipped-stage / skip-planned / restart-current / restart-previous (cascades downstream) / change-depth (re-runs completed stages at the new rigor — partial completion is not a shortcut) / pause / change-architecture (full cascade, most expensive) / add-remove decomposition units. Every mutation op gets an engine-computed “what re-runs / what stays” preview; the user confirms the cascade, not the op. After a mutation re-runs nodes, downstream nodes whose inputs changed but weren’t re-run get a cheap consistency re-validation pass, escalating to re-execute on mismatch. “Never silently mutate.”
LLM-authored mutation batch rules (distilled from documented failure modes): ops use unique-anchor XOR positional addressing, never mixed in one batch; no two ops target the same node (batch rejected with diagnostic); apply in reverse-index order so earlier coordinates stay valid; normalize common kind/field aliases; atomic failure contract — a rejected batch writes nothing; chat-tool mutation flows enforce staged turns where an inspect op echoes the current spec into the model’s context so it mutates what it just saw, not what it remembers.
Version snapshot on every mutation (node-granular patch ops auto-snapshot), and re-verify the content revision after approval and immediately before apply — the TOCTOU gap approval flows otherwise have.
Replan emits only remaining steps: a mid-run replan asks the planner for “ONLY the remaining steps” given the critique, resetting a local index while a monotonic global index keeps journal keys dense — completed epochs untouched.
Escalate-and-reclassify is a mutation, not an abandonment: mid-run discovery of higher risk maps to a typed op (“upgrade tier: splice skipped stages ahead of the frontier”).
Spec-drift detection: tested_signature — hash the spec at last successful validation/run; mismatch surfaces a “re-validate first” warning. Cheap and effective.

Budgets

Budget object per run and per node: {max_tokens?, max_cost?, max_duration?, max_steps?}, controller-enforced (never prompt-enforced), warn at ~80%, and land gracefully: on exhaustion inject a wrap-up hint and set tool_choice=none for one final synthesis step — the agent must produce an answer — then hard-stop. Budget state keyed per (enforcer, run) so it survives HITL parks.
Budget pre-charge on resume (above) and cost caps checked before claiming the run slot — a capped workflow records a skipped run and can’t wedge its next fire; monthly spend = sum of run costs over a 30-day window; listings project cost_estimate = last run cost × fires_in_window.
Model-tier portability: encode intent (smol|regular|smart|ultra or cheap/deep tiers) on nodes/templates rather than provider-coupled model IDs; reserve the expensive tier for judge/decision nodes — one measured system runs all workers on the cheap tier and exactly two judges on the deep tier.
Equal-budget competitive runs as an evaluation shape: same task, N models/templates, identical budgets, per-round metric snapshots, full decision persistence (prompt, raw response, parse status, tokens, latency), halting on consecutive failures (verification-and-judging).

Run admission, genealogy, readiness

One active run per (definition, container) with a 409-style refusal recorded as a skipped run with a reason — observable, not silent. Distinguish the two resume kinds (wake vs gate-answer, above).
Run genealogy schema: root_run_id, parent_run_id, spawned_by_step_id, join_group_id, branch_key/branch_index, with a (root_run_id, status) index for O(1) tree queries; spawning a child emits a child_run_attach event so UIs attach to nested streams. Depth guards on nested invocation are mandatory — one system shipped agent-to-agent invocation with no depth limit; another banned workflow nesting outright as the simplicity trade-off.
Readiness preflight: templates aggregate one-hop child requirements (binaries on PATH, packages, credentials); missing credentials block start rather than silently degrading mid-run. A dry-run readiness verdict (ready/warning/blocked + next_actions) belongs at plan-review time.
Triage gate as the canonical first node: a classification whose output selects among 2–3 entry subgraphs and skips declared node ranges — every serious workflow in a 7-workflow corpus opens this way — plus baseline-capture (run the validation suite before the first mutating node; later gates diff regression vs pre-existing).

Patterns & compositions

The five canonical workflow patterns (industry-shared vocabulary; name compositions after them): prompt chaining (sequence + programmatic gates), routing (classify → dispatch, incl. model-tier routing: easy → cheap model), parallelization in two variants — sectioning (independent subtasks) and voting (same task N times + threshold), orchestrator-workers (runtime decomposition → foreach → synthesize), evaluator-optimizer (generator + critic loop, when criteria are clear and iteration measurably helps).
Produce-and-audit / fact-ledger→synthesize→audit: a mid-graph ledger node records provided context, verified data, UNKNOWN_DETAILS, FORBIDDEN_INFERENCES; synthesis binds to it; a final read-only audit node enforces the ledger and is pinned as final_output. Anti-hallucination as graph structure.
Debate as a compilable macro: shared-history string + per-side histories + count threshold (k × rounds) + last-speaker routing + a judge node — expressible as loop + transform + gate, parameterized by roles[] and rounds; durable inter-node context is typed reports, free NL lives only inside the bounded debate scope whose outcome is written back structured.
Drain loop: decompose into small stacked units (each branched off the previous), then a bottom-up verify/merge loop that halts on first failure so a broken base can’t poison the stack; maxIterations capped.
No-op-first recurring workflow: node 1 of every recurring workflow is a cheap “did anything change?” check; the no-op path is genuinely silent; explicitly distinguish “nothing changed” (silent success) from “I failed to check” (real failure that alerts).
Template generation = pattern-pick + slot-fill, not freeform spec-writing: classify intent into a small library of proven shapes (sequential / fan-out-merge / condition-gated / monitor-synthesize), deterministically fill slots, then run a gate ladder (collision check → lint → risk classify → smoke → E2E → baseline acceptance: the candidate must beat a single-model one-pass baseline at a hard threshold, e.g. ≥0.80 weighted score, or it doesn’t persist). See planning-and-decomposition.
Plan-as-artifact + approval-as-handoff: the plan is a markdown file annotated with inline comments; the approve dialog simultaneously picks autonomy mode, executor/model, and run environment. Planner and implementer may differ.
Externally-driven execution mode: the degenerate but useful mode where the engine only tracks state (pull-based frontier protocol) and the chat LLM or a human performs each step.
Codified template-to-template hand-off edges (incident→bugfix, bugfix→feature, cleanup→refactor…): declared graph edges between workflow templates, with a required statement of what was found and why the current workflow no longer fits.

Anti-patterns & failure modes

Schema without semantics: shipping checkpoint/retry/parallel-level fields the executor ignores. Dead fields mislead every future reader; behavioral acceptance tests are the cure.
Silent-empty-string binding resolution: unmapped/missing references interpolating as "" produce garbage prompts that run. Fail loudly with typed BindingError + journaled diagnostics.
Whole-value-only expressions: no filters/interpolation → comparison logic leaks into node executors as regex hacks.
Direct status writes: any path that can set run/task status other than the derived projection eventually overwrites a user stop with “success”.
Auto-catch-up of missed fires: replaying every elapsed schedule slot after downtime causes storms. The working design: enumerate capped (~480), keep newest ~20 as a review card, collapse the rest into one summarizing skipped record, roll next-fire forward (automation-and-triggers).
No overlap/double-fire guard: a scheduler with no in-flight check and no fire-time lock is a proven double-fire hazard; also cooldowns keyed only on last-success never engage when every run fails — key cooldown on last-attempt-start.
Timestamp-only unique names: second-granularity names collide for parallel spawns in one tick; include randomness.
Non-idempotent side effects of deduplicated writes (the counter-inflation bug) — replay corrupts salience/metadata silently.
Killing the process on completion-signal sight: loses trailing token-usage and structured-output events; use the post-signal grace window instead.
Tolerant/forgiving output parsing: hides genuine model failures; strict JSON with fence-unwrapping only.
Compaction as the stage-boundary strategy: agent-generated summaries inherit the agent’s biases and rationalizations; clear, don’t compact — every gate produces a durable artifact and the next stage starts from artifacts only. “If you want compaction, your gate didn’t produce a sufficient artifact.”
Checkpoint-not-presentation: a complete-looking artifact mid-workflow is evidence the workflow ran, not that gates passed; presenting final artifacts before gates biases acceptance.
Retrying/degrading prompt assembly (see principle): a degraded prompt burns an iteration and possibly commits garbage — clean abort with typed diagnostics.
Nesting without depth bounds: recursive workflow/agent invocation shipped without a depth counter in one system; another banned nesting entirely. Either bound depth explicitly or ban it — never leave it implicit.
A broken event subscriber killing the run: all observability callbacks/broadcasts must swallow and log their own errors.
Journal cuts that orphan tool results — API-rejected conversations on resume (see dangling-tool-result invariant).

Quantitative findings

Idle timeout default 600s (reset per output event); post-completion-signal grace window 60s → succeed with warning, not fail; per-deterministic-node timeout default 30s; sandboxed wait-filter predicates: 50ms wall-clock, 2KB source.
Unattended approval timeout 30s vs 600s attended; HITL clarify-form timeouts 24–48h.
Circuit breakers: ~100 node executions/walk, ~500 steps/run (checked at every walk entry); production implementer loops run fine at maxIterations: 100 on just max-iterations + completion-signal + dual timeouts.
Loop autopilot thresholds: auto-pause at 3 consecutive failed cycles; “stuck” at 4 byte-identical evaluation outputs; duration anomaly at 5× rolling average; relaxed rubric ratchet tolerates regression to previous × 0.9; agent-proposed wake default 300s.
Scheduling: recurrence anchored to creation time (not re-phased to “now”); persist next-fire before executing; 15-min minimum interval floor; ran_late = start >300s after slot; missed-fire enumeration cap 480, newest 20 reviewable; single scheduler loop sleeping clamp(next_due, 1s, 60s).
Budgets/economics: ~10¢/task ≈ 30–50K tokens (workflow territory); an agent step’s working context is ~10–20K tokens; 5× token overspend at 1M tasks/month ≈ $1.5M/yr; generated-template acceptance threshold ≥0.80 vs one-pass baseline (weights 40% usefulness / 25% specificity / 20% gate-risk-collision coverage / 15% reusability).
Storage: per-definition run trim 200; JSONL diff-audit soft cap 256KiB → truncate to ~128KiB; run-history reads O(history) via per-definition files; context-preamble chain cap 2000 tokens (immediate parent inline, ancestors as pointers).
CI/verification watch loops: poll 20s, timeout 1800s, 60s no-checks grace, 20s settle window so late-registering checks can’t fake green; repair attempts = max(3, repair_rounds+1).
Evaluation methodology for workflow-vs-prompt claims: ≥30 tasks, k≥3 runs per (task, condition), fresh context per run; metrics TPR / RP@k / regression-free rate / clean-pass rate; two blinded raters + adjudicator, κ<0.6 downgrades to exploratory; power warning if CI half-width >±0.15; validation_passed=false if any check was added/removed/disabled/skipped/modified; would-be-reverted work counts as failed, not rework.
Coalescing windows for high-frequency event streams: 50–250ms; per-producer rate floors survive even forced/manual refresh.
One engine’s whole graph walker with joins, cycles, checkpoints, and recovery fits in ~750 lines; a budgeted propose-evaluate optimization engine in ~430 — the core is small; the semantics are the work.

Open questions

Where does the graph-vs-agentic equilibrium settle? One camp deleted its graph layer for tool-driven coordination; another ships graphs with agentic escape hatches. Unresolved: whether typed mid-flight mutation + cheap NL→spec generation is enough to keep declarative engines competitive as models improve, or whether graphs retreat to the high-reliability/unattended niche.
Budget semantics are named but not standardized: budgets in tokens, money, wall-time, and iterations interact (which trips first? do child runs inherit or partition the parent budget?); no source defines composition rules for nested runs.
Async inter-run communication: sibling data bindings exist; a mailbox/handoff primitive letting concurrent runs exchange messages asynchronously (beyond parent-child sync returns) is a named future direction with no shipped design in the corpus.
Compensation is under-specified: the effect ledger records compensated and provisioning contracts require idempotent teardown, but no corpus system ships general compensation orchestration (saga-style) for rewinds crossing committed external effects.
Distributed/multi-node execution: every serious engine in the corpus is single-process (SQLite/JSON files + pollers); the Redis-bus + wakeup-dispatcher shape points the way but was flagged WIP even where designed.
Parallelism remains the most-designed, least-shipped feature: multiple engines computed parallel-safe levels and then executed sequentially — what minimal executor change makes level-parallel execution safe against the join/active-edge subtleties by construction?