Knowledge & Memory

Two related but distinct subsystems: Knowledge is the user’s ingested content library (notes, documents, media) with a processing pipeline and hybrid search; Memory is what the assistant learns and recalls across conversations. Paths are relative to PersonalClaw/src/personalclaw/.

Knowledge

Store

knowledge/store.py — knowledge.db (SQLite) holding items, an FTS index, and embedding vectors. The raw embedding vector never leaves the DB — API responses carry only a has_embedding flag. Deleting an item cleans vectors, mentions, and FTS rows — not just the item row. FTS values are kept in sync on update (delete-with-old-values, insert-with-new).

Another home’s library comes into this one row by row, whichever way it arrives: a merge restore, an archive import in merge mode (Settings → Import / Export) and a folder sync all run snapshot._merge_sqlite_attach, so the other copy’s items, watched sources, collections and tags come in and this home’s stay. A library numbers its tags itself, so another library’s tag 1 is not this one’s: tags come in by name, and each item stays filed under the tags it had (durability.numbered_rows). An item keeps its document and thumbnail as paths into the library’s files folder under the home that wrote it; once an import or a merge restore has brought the files in, and whenever the library opens in another home, each such path is pointed at this library’s copy of the file (knowledge.arrivals.repoint_moved_documents), and only ever at a regular file inside its own folder that no other item keeps. The source row the system makes for the artifact mirror has its provider’s id in every home (knowledge.arrivals.system_source_id), so a merged library holds one; a library holding another for it, from an earlier merge, folds it in when it opens; of two mirrors of one artifact the one the provider’s row holds stays, and the artifact’s next save brings it up to date (knowledge.arrivals.fold_system_sources).

Project scoping

Knowledge stays ONE global library — a project is a tag plus item metadata, never a second database. knowledge/project_scope.py owns that scoping for the items a workflow run writes: project_id (the producing container, first writer wins), run_id, and a closed sharing_policy (private | shared, default private). A project’s Knowledge view shows its own items whatever their policy plus other projects’ shared items labeled with their source project; another project’s private items never appear. The project tag is the same one knowledge/session_brief.py::project_tag reads for a run’s project brief.

Sharing a knowledge item OUT (the push half)

sharing_policy also decides what leaves the machine. knowledge/sharing.py owns the outbound half and is the ONE place the gate lives: a shared item is offered to every registered provider’s KnowledgeProvider.push — the mirror of ingest — while a private item never reaches a provider at all, so no provider can widen the default. A provider with no outbound half returns the ABC’s None and is reported as having declined, never as a delivery.

What crosses is an allowlist of four attribution keys (contributor plus the scope trio), not the item’s whole file_metadata: an item’s claims and citations are local bookkeeping. contributor is the same key vector_memory attributes semantic memory with, rendered by the same identity.contributor_label.

Coming back, SourceEngine carries those four keys onto the row it writes and files it under its container’s tag, so a teammate’s contribution reaches both readers labelled: the project Knowledge view names them, and the session brief puts the contributor on the fence’s source label inside fence_untrusted (labelled and fenced — shared-store-provider-conformance.md clause 2). Only foreign contributions are labelled; labelling the owner’s own rows would hide the one case the label exists for. A push never fails the local write.

An upload landing

A file arrives by one request (POST /api/knowledge/ingest) or by the resumable protocol (uploads/store.py: init, parts, complete). Nothing whose cost grows with the file runs on the event loop, where on a 512 MB upload it stopped every other request for up to 0.34 s:

• a request body is written to disk through uploads.spool.Spool, a megabyte at a time in a worker thread (on a busy disk one write of the next chunk took 0.1 s);
• a complete assembles the parts, and the item’s file is moved (a copy across disks), hashed and thumbnailed, in worker threads; file reads and writes and the hash leave the interpreter lock;
• the content scan (below) runs in a child process (uploads/scan_child.py, started by uploads/content_scan.py), because its parse holds the lock; the child imports the scanner and nothing else of the package, since every upload, Knowledge write and artifact save waits for its start;
• what the file becomes is decided in one worker call (knowledge.file_items._decide): the duplicate check, the insert and the read of the new item, one upload at a time in the process and under the store connection’s lock (SharedConnection.held), so two uploads of the same bytes still make one item, even decided on two stores over one library. A change the store makes of several statements holds that lock from its BEGIN to its end, so the decision never starts inside another thread’s open change. A commit that waits on a busy disk (0.2 s, measured) no longer stops the event loop itself, but loop code that uses the same store waits for the lock meanwhile, among it the Knowledge list and its poll, the tags, and the ingest runner’s reads and writes. Most Knowledge routes and ingest stages still call the store on the loop; moving them off is the follow-up;
• the module that makes the item is imported with the upload handlers, so no complete pays for its first import on the loop (0.13-0.44 s on a busy machine).

While an upload’s complete runs, UploadStore.completing holds it: a retried complete, a late part and a drop are refused with 409 (upload_completing for the drop), and the sweep passes it over. tests/rails/test_event_loop_whole_file_census.py keeps whole-file work off the loop everywhere in the package, and names what is still on it.

The content scan

Every route that stores an uploaded file the agent or the library will later read hands it to uploads.content_scan.scan_upload before anything is made from it, whether it came in one request or in parts: a chat attachment, a file uploaded to a folder, a Knowledge file, a file dropped into a workflow run, a binary artifact’s new bytes, a pinned screen frame, a project archive and a backup import. Two doors bring a file in no request and are held to the same checks: a file dropped in the memory vault’s raw/ folder, taken as a Knowledge file is when a sync sweeps it (knowledge.file_items.take_file), and a file in a folder Knowledge watches, taken when a poll finds it new or changed (knowledge.file_items.take_watched_file). The same kinds, the same size limits and the same scan, each refusal filed as a failed item that says why. Neither takes a file that is still being written: its size and its times must hold still across a two-second settle window (file_items.still, one window for every file of a pass), and a copy that changes while it is made is not kept; the file is left for the next pass, rather than taken as the part of it copied in so far.

The scan reads an upload by its bytes, whatever its name or its declared type says it is, with both of the scanner’s surfaces, the destructive-script rules and the prose-injection rules, as two windows at most: its first 256 KB and its last. A file of up to 512 KB is read whole, so no byte of it goes unread: its last window is the rest of it, read on from the first. A larger file gets the large-file policy, and what lies between its two windows is not read. A window that holds a NUL byte is binary and is not read (random runs of binary bytes read as false alarms), while the file’s other window still is. So an ordinary picture, recording, video or archive, whose bytes are binary, is never read, and an SVG drawing, which is text, is read like any text file. An archive is scanned as the file it is.

What a reader makes of an upload is scanned as well (scan_text), because the bytes do not show everything a model is handed: the text beside a stray NUL byte sits in a window the byte scan skips, and a PDF’s or an Office document’s text sits in compressed parts of the file. The text the document reader makes of an upload is read by the same rules before a model is handed it or it is kept for one, except that no window of it is skipped for a NUL byte: a chat or an Inbox attachment’s text when it is extracted, a Knowledge document’s when it is ingested (refused, the item keeps nothing of it and its status says why), a code file’s when it is uploaded to Knowledge, and a document’s that the agent opens with read_file. What a model writes of a picture, a recording or a scanned page (OCR, a description, a transcript) is not scanned. A chat turn hands the model no more of a long text than the scan read of it. The readers agree on what text is: a file whose first 8 KB hold a NUL byte is binary to every one of them (the Files view’s own test, file_view.is_binary), and any other is read as UTF-8 with a byte it cannot read marked, never guessed at, so a zip or a program is not read as text. A file that opens with a UTF-16 or UTF-32 byte-order mark is read in the encoding the mark names. That text reaches the model fenced as data (security.fence_untrusted), as a fetched page does: an attachment’s text in a chat turn, a referenced Knowledge item, and a document read with read_file.

Text from outside that the library keeps with no file is read by the same rules before it is stored (knowledge.text_items): a watched source’s entries (see Watched sources, below), the page or paper a bookmark’s scraper fetches (scanned before the ingest keeps any of it), a web watch’s new items (triggers/web_poll.py), a note an app writes or edits through /api/knowledge/items, one the agent writes with knowledge_create, knowledge_update or log_decision, one a workflow’s knowledge-persist step writes, and an edit made to a page of the knowledge vault (knowledge/vault.py), which is a file. A door with someone to answer is told in the scan’s words and nothing is written: “Its text failed the content safety scan, so nothing was made from it.” for a new note, ”…, so the item was not changed.” for an edit. A source’s entry or a web watch’s item the scan refuses is a failed item that keeps no text and says why, named in the gateway log; the vault leaves the note as it was and stamps why into the page (sync_conflict). A note the owner writes in the app is her own words, as her chat message is, and is not scanned.

An artifact’s text reaches the library through its mirror (knowledge/artifact_ingest.py), so it is read by the same rules where it is written, as what the mirror keeps of it: its name, its description and its body’s words, for a kind the mirror keeps (artifact_ingest.text_refusal). The doors are the agent’s artifact_save, artifact_update and a csv from sheet_create, an app’s POST or PATCH /api/artifacts, and a workflow’s publish:, its artifact-update step and its render-report step; each answers in the scan’s words and writes nothing, so no mirror holds the text. An artifact the owner saves or edits in the app is her own words and is not scanned. An artifact that points at a file holds the file’s text, which any program can write, so the mirror scans it before keeping it whoever saved the artifact (ArtifactIndexer._mirror_file, a task when the write ran on the event loop): what the scan refuses, or could not check, is a mirror that keeps no text, named by the artifact’s slug, says why and is said in the gateway log; one it could not check is read again the next time the mirror reads the artifact.

Refused content is answered 422 upload_content_refused, and nothing is made from it. A scan that did not run is never read as a pass: a window that cannot be read, a child that cannot start, one silent past 60 s, and one that ends without an answer (as it does when the scanner raises on the content) are answered 503 upload_content_unchecked; text a reader made that the scan refuses or could not check is withheld in the same way. Each refusal is a row in the security event log (upload_scan, naming the route, or where the text was going). tests/rails/test_stored_upload_scan_census.py reads the package for every route that takes a file’s bytes from a request, and fails on one that neither scans them nor says why not.

The artifact mirror follows every write, whichever process made it

Knowledge keeps a copy of each artifact of a kind it reads (knowledge/artifact_ingest.py), found by a search and never listed. It follows the artifact store’s writes rather than polling the store: each write method of the store calls artifacts.changes.emit once its write has landed, and the gateway subscribes two observers there when it starts, the mirror and the hint that tells every open page to read the artifacts it shows again (DashboardState.announce_artifact_change).

The store has writers outside the gateway: an agent CLI’s artifact tools run in the tool server the CLI starts (personalclaw mcp-core), a process of its own. A process that is not the gateway of its home (the gateway’s runtime record names its pid, gateway_base.live_gateway) tells the gateway the slug of each artifact it writes: POST /api/artifacts/{slug}/changed, with the internal credential and the chat the work is for. The gateway reads the artifact from its own store and tells both observers what the store holds now: written when it holds the artifact, removed when it does not. The gateway never tells itself, so each write is heard once, wherever it was made. The call carries no text, so the content scan reads an artifact’s text once, at the door that writes it, in the process that writes it. The gateway tells its observers as the work of the chat the call names (dashboard/memory_write_gate), so what Knowledge keeps follows that chat’s mode, as when the gateway’s own agent writes. An Incognito or Temporary chat’s agent writes nothing there at either door: its artifact tools change nothing in the library (mcp_artifacts). A process with no live gateway of its home tells no one, and a gateway that cannot be told leaves the write as it is and is named, with the artifact, in gateway.log.

Ingestion pipeline (node graphs)

knowledge/pipeline/ is a node-graph executor:

• graphs.py maps each of the 12 native item types to a code-owned PipelineGraph subclass. Users tune per-node execution parameters (enable/backend/use-case/timeout) via config but cannot rewire a graph.
  • Text types (note, gist, journal, fleeting) → PassthroughGraph (the content is the extracted text). A code or config file is a gist, whose text the library reads into the item when it is uploaded. A chat or an Inbox attachment is only the file, with no content to pass through, so extract_file reads a file of a text type with DocumentGraph (graphs.file_graph_for): its reader reads the file as text by the shared rule, and the scan reads that text.
  • bookmark → BookmarkGraph (scrape the URL; user-pasted content passes through without a fetch).
  • Document types (pdf, document, sheet, slides) → DocumentGraph (pure-python file read → consolidate).
  • Media types (image, audio, video) → media graphs (ImageGraph runs exif ∥ ocr + vision); model-backed nodes degrade gracefully — no bound vision model means the node is skipped, never a hard failure.
  • Every step records what became of it (pipeline/outcomes.py), and the runner persists the map as file_metadata.node_phases: {status, reason, fix, needs} per graph node and terminal stage. done; failed (the reason is the error); skipped — it could not run for want of something the owner can add, with the fix (each {text, href}, an in-app route) and the capability it needs (a use case, or ocr_engine); not_applicable — it does not apply to this item (a branch not taken, no intents to match, a no-AI source), with no fix. A step that waited on a skipped one is skipped too and carries that one’s reason and fix: a video with no Image · Modality model says why its classifier, OCR and Vision did not run, each on its own line. An ocr step that a model OR an installed OCR app could have run names both. The item-graph read adds ready to a skipped step whose need is there now, asked of the probes the executor asks (for Image · Modality, of its binding, so the read never waits on a catalog listing), and the item page then offers to run the item again. The item’s status line (processing_error) carries only what went wrong.
  • A bound model that cannot answer fails its step, with the reason. transcribe_audio(_detailed) return a transcript (empty text = no speech) or raise SttError with the sentence saying why there is none; diarize_audio returns speaker turns ([] = no one spoke) or raises DiarizationError. The transcription and diarization nodes turn either error into a failed step whose sentence is the item’s status line, so a recording is never “done” without its transcript. A recording with no speech is done, and says so: the runner promotes no_speech onto file_metadata, which the detail view shows beside the step. Transcription and diarization get the duration-scaled node budget, and a budget that runs out says how long it was.
  • A node’s budget is on its own time (cancellation.wait_for_unpaused): time the event loop could not run is not counted. A wall-clock bound failed a frame extraction whose ffmpeg finished in 13 seconds as “did not finish within 2 minutes”, because another step’s engine had frozen the process. A node out of its budget is cancelled, and the cancel kills the program it started (media_nodes._run_cmd’s ffmpeg, a model’s child process); the executor logs one line naming the node and the item, and the step fails with retry, so the item page offers to run the item again. A node’s own TimeoutError (a request inside it that timed out) is its failure, not its budget’s. A run cancelled from outside (the gateway stopping) says which node it stopped.
  • A video is seen across its whole length. frame_extract takes up to 8 frames, one in the middle of each of 8 equal slots of the length ffprobe reads (a slot whose middle has no frame to seek to takes the one at its start), and each denser pass the classifier asks for adds frames between them. The steps that read frames (video_classify, vision, and ocr for a text-heavy video) take theirs spread across all of them, in time order; ocr reads an image or a page on its own. The runner promotes frames_sampled, frame_times and video_seconds onto file_metadata, and the detail view says how many frames were taken across how long, and when. A video whose length cannot be read gets a frame every 10 seconds from its start, and the view says that instead.
  • An item’s text is what its graph’s last step made (_item_text): consolidate for a document or an image, video_consolidate for a video (it reads the narration through lexicon_correction, the last step that worked on it), and lexicon_correction for a recording, whose text keeps the speaker labels speaker_fusion put on it and the corrected words. When that step made no text (skipped for want of a model), the item gets the texts it would have brought together, in step order. A recording in which fewer than two people speak carries no labels.
• Terminal stages are not graph nodes: after a graph completes, pipeline/runner.py runs consolidate-pool → insights → chunk+embed once over the whole extracted-content bundle (they operate on the item bundle, not a single node’s input). Embed and the dedup after it run in a worker thread, as the embedding re-index does, and so does the ingest queue’s building of each item’s embedder (create_embedder_from_config asks a bound model for one vector, to learn its width): a bound model’s answer is waited for in the calling thread (run_embed_sync), so on the event loop each answer stopped every request the gateway had.
• knowledge/insights.py produces {summary, key_points, topics, action_items}; entity and intent extraction follow. The AI title replaces only a placeholder no person set — an upload’s file name, a note’s opening words. A title a person set (typed at creation, kept in the create form, the file’s own name included, or edited later) is recorded as hers (items.title_source = 'user') and never replaced; the suggestion stays on the item as ai_title. A bookmark’s scraped page title is held to the same rule: it replaces only the URL a bookmark is titled with until its page is known, never a title a person set, and stays on the item as url_title. What the graph’s results fill on the item (that title, the text of an item that had none, the structural metadata) is decided from the item as it is once the graph and the content scan are done (ingest_item reads it again), so a title or text typed while a page was fetched or a file read is kept.
• What comes in as HTML is stored as its words. The library renders a body as markdown only (the web app’s one renderer shows embedded HTML as text — see security.md), so every way HTML comes in converts it to markdown text, and no raw HTML survives outside code: a tag the source only showed as text comes out of html2text or trafilatura as a real tag, so it is written back as text (connectors/base.py::without_raw_html). A <pre> comes out as a fenced block, which that step knows for code.
  • a watched feed’s or page’s entry — connectors/base.py::readable_text;
  • a bookmark’s scrape — connectors/web_url.py;
  • an uploaded .html file, a watched folder’s .html file and an html artifact’s mirror — readers.py::html_to_prose;
  • a document artifact’s mirror — its editorial HTML through the same reader, unless the body has no HTML block structure, in which case it is the markdown it was saved as (artifact_ingest.py::_reads_as).
• What was stored as markup before is converted once. A watched feed’s or page’s item that still holds raw HTML outside code is converted at gateway start, a batch at a time (knowledge/stored_markup.py): its pool copy follows, and its chunks and vector are invalidated for the maintenance host to rebuild, with no model call. A document/html artifact mirror that still holds raw HTML is re-mirrored at start (ArtifactIndexer.remirror_markup). Both key on the stored text, so a converted body never matches again. A body a person wrote is never rewritten: any HTML in her note is shown as text.
• One reader at a time, hers first. knowledge/ingest_queue.py runs items one at a time (the terminal stages share one sqlite connection) in two lanes: what a person adds or asks for (enqueue) is read before background work — a watched source’s items, a whole-library regenerate, the vault and artifact mirrors (enqueue_background). Nothing preempts the item being read, so her wait is at most that one item. standing() says where an item is — how many are ahead, which lane, and the median of recent items’ times once three have finished — and the item and list reads carry it as queue.
• Readers (knowledge/readers.py) cover the 12 create formats; knowledge/connectors/web_url.py fetches bookmark/URL content through the egress chokepoint (net_fetch with egress_policy_for(CONNECTOR) — see security.md); knowledge/dedup.py deduplicates; knowledge/llm_pool.py pools background LLM workers.

Embedding

knowledge/embedder.py — UnifiedEmbedder, the one provider-agnostic embedding path: it wraps embedding_providers/registry.py::get_active_embed_fn(), which resolves the embedding use-case binding (Settings → Models). Nothing bound → embeddings are gracefully off (no crash; vector search simply doesn’t participate) — but gracefully off is not silently off: an item ingested with nothing bound wrote no vector and no chunk, so it is recorded unsearchable rather than done (see Searchability). Any provider works: the native apps/sentence-transformers app or any bound remote model.

Every embedding call that reaches a model is recorded with the model calls (model_calls.jsonl, use case embedding): the model, how many texts and about how many tokens (estimated from their length, and marked so), how long it took, what it cost, whether it answered, and the session the work was for. Never the text. One that answered also writes its Settings → Usage row (usage_ledger.record_call), which names that model-call row so the two are not counted twice. The call is priced by the one pricing function (routing.rates.price_call: a price you set, a model on this machine at its known $0, the shipped table); one that fails is charged nothing and is in the model-call log only, as any failed model call is.

Every vector records the model that wrote it — a chunk’s and an item’s alike (embedding_model_id / embedding_provider, knowledge/embedding_fingerprint.py) — so the re-index Settings → Models starts after an embedding change re-embeds only what the model bound now has not: it clears the other items’ vectors first (so no search or dedup compares one with the new model’s meanwhile), then embeds those and the ones never embedded (KnowledgeStore.clear_stale_embeddings + reembed_all(only_missing=True)), and re-embeds the stale chunks. An item embedded before items recorded their model names none and is stale, so the first start after that update re-embeds each once.

It is the one re-index, and every surface reaches it the same way. The gateway’s start runs it for what the bound model has not embedded: items, and the passages a stop left after the items (_pending_reembed counts both), each at the width the model writes. Settings → Models and the Knowledge page’s embedding chip read the running job (GET /api/models/embedding/reindex) and follow its progress through web/src/lib/useEmbeddingReindex.ts, and the chip’s re-embed starts or joins it (POST /api/models/embedding/reindex answers the running job rather than a second). Its knowledge half is resumable because its backlog is derived from the rows: a cleared item holds no vector and a stale passage records another model, so the next start finds both. A stop anywhere also leaves no passage whose row names the new model while a copy of its vector still holds the previous model’s: each item’s re-embedded passages go to the ANN index and the bound external store first and to their rows last (KnowledgeStore._write_reembedded_chunks), so what a stop cuts off still reads stale and the next pass writes it again.

Watched sources

knowledge/source_engine.py polls each source’s provider on its interval and persists what it offers through the source’s novelty gate (source_seen), advancing the cursor only after every item is durable.

• Poll floors. A source that fetches is never polled faster than sources.network_floor_secs; only core’s own folder observer (DirSourceProvider, matched by exact type, so an app’s subclass stays under the floor) is held to LOCAL_FLOOR_SECS instead. effective_interval is the one computation, and the sources list shows it as poll_every_secs.
• The per-poll cap (sources.max_items_per_poll) counts what a poll indexed, not the first N sightings offered, and a poll the cap cut short whose sightings are all first sightings keeps its cursor, so the next poll takes the next ones. A provider whose cursor must move asks for the cap (max_items, one of ENGINE_POLL_KWARGS) and stops at it.
• A watched folder’s file is taken as an upload is. The folder observer (DirSourceProvider) reads no file: a settled sighting names the file inside the folder (file_of, asked again when it is taken), and the engine takes it through the upload door (source_engine.takes_files, the folder observer by its exact type; knowledge.file_items.take_watched_file). It is typed by its kind (media.classify), held to the size the upload policy allows it, a private copy of its bytes scanned, then kept in the library’s own files and read by the reader for its kind when it is ingested: a document’s text, which the ingest scans as well; a script’s code, read and scanned as it is kept; a picture by its graph. What a check refuses is a failed item that says why and keeps no text and no file; a later change that passes makes it again. No file is decoded into a note: a kind the library does not take is refused, and a reader that finds no text in a file says so. A change remakes the item the folder made of the file, never a second one, and a file whose time moved but whose bytes did not is not read again. A folder set to no AI has its files read by the readers that need no model (RawDocumentGraph, RawImageGraph). A file still being written is left for the next pass (see the content scan, above). A note an earlier version made of a watched file is left as it was until the file next changes; to have one read the new way now, save its file again.
• Any other sighting’s text is scanned before it is kept. A feed’s entry, a page’s, and what an app’s source hands in (a repository’s files, a shared store’s items) carry their text, and the engine scans it (text_items.refusal) before create_typed_item writes anything. What the scan refuses, or could not check, is a failed item named by its link or its guid that keeps no text, said in the gateway log. A sighting the source has had is not read again (KnowledgeStore.source_has_seen, the novelty gate’s question asked before the scan), except one the scan could not check, which is read when the source offers it again. A change the source reports is read when its text moved (file_metadata.content_hash); refused, the item keeps none of what it held. A watched folder states no item_type: its files are kept as their own kinds, so its kind declares no default_item_type and a create that names one is refused.
• A watched folder’s first scan reads in what is already there, newest first, up to FIRST_SCAN_MAX_FILES files or FIRST_SCAN_MAX_BYTES, each file counted whole; a file too large for what is left of the bytes is left out on the way, and the scan carries on with the older ones. What it left out is recorded as seen and comes in when it changes. It stops at the cap and counts what is still to come, and the sources list shows the scan (first_scan: found, left out, waiting).
• A watched folder takes in only what is inside it. Each path is taken in under the path it really is, links resolved first (dir_source.resolve_in, then takes, the rule the agent’s file tools ask of the same folder): a link to a file or folder outside the folder is left out, and the sources list counts them (links_outside); a link to a file inside comes in once, as that file. A note an earlier scan took in through a link out of the folder is removed at the next scan with its sighting (forget_source_item), not archived.

Search

knowledge/retrieval.py — HybridRetriever: FTS5 keyword + graph traversal + optional vector search, fused with reciprocal-rank fusion (RRF). A minimum cosine floor keeps weak vector hits from polluting precise keyword queries. search() returns hits only; search_with_diagnostics() returns the same hits plus the typed reasons the library could not answer.

The vector arm’s candidates come from a sqlite-vec index of the chunk vectors (knowledge/vector_index.py), in the same database file. Rows that arrive around the store’s write-through, a merge restore, an archive import or a folder sync taking another copy’s chunks, rebuild it from the chunk rows (ChunkVectorIndex.rebuild_all, from snapshot._merge_sqlite_attach), and the full-text index is rebuilt by its own command; a process that cannot load sqlite-vec leaves the index to the store’s reconciliation on its next search.

Searchability

knowledge/searchability.py owns ONE vocabulary for “this item persisted and search cannot fully reach it”. Two failures used to persist as processing_status: "done" with no error: an image-only PDF, where document_read reported success and extracted no text (leaving only the synthesized structural descriptor — none of the document’s words), and any document ingested with no embedding provider bound (zero rows in chunks, no item vector).

• The named status is unsearchable — a distinct value, because partial already means “an OPTIONAL step was skipped or failed” and is often benign. The token is not the claim: for every reason except no_extractable_text the item’s text is indexed, so keyword search reaches it and only semantic search cannot.
• The typed reasons are closed: no_extractable_text, no_embedding_provider, not_indexed, plus two read-time ones, which no ingest records because what changed is the bound model: stale_index (vectors from a different embedding model than the one bound now, or with no model recorded — a passage’s or the whole-item one) and awaiting_embedding (an item recorded no_embedding_provider that still has no vector while a model is bound now: it was saved before the model was, and Maintenance’s embedding job or the re-index embeds it).
• One sentence per reason, minted once. Degradation.summary is the count-bearing claim (“2 items and 1 artifact have no embeddings because no embedding model is bound — keyword search finds them, semantic search cannot”); the Doctor row and the knowledge_search note print it verbatim. The item’s status line reads reason_detail() — a sentence, never the token, which stays in file_metadata.unsearchable_reason for machines.
• Counts are the library’s. A row carries the shelf it is shown on: an artifact’s search mirror and a report’s finding are indexed but never listed, so they are counted apart under their own nouns rather than as “items”.
• The verdict is computed from what LANDED — the item’s rows in chunks, its embedding column, its stored text — never from a stage’s self-report, since the self-reports are what were untrustworthy.
• pipeline/runner.py persists the status plus the reason at file_metadata.unsearchable_reason; a re-ingest that lands clears both, and so does a vector landing any other way — the re-index or a backfill (KnowledgeStore.retire_embedding_verdicts, from replace_chunks and reembed_all), so an embedded item stops saying it has no embeddings. The store retires, as it opens, any verdict an item’s own vectors already make false (settle_embedding_verdicts): a re-index from before the writers retired it left some standing, and nothing else revisits an item that has a vector.
• Two surfaces READ that one recorded fact rather than re-deriving it, so they cannot drift: the knowledge.searchability Doctor probe (one row per affected item) and knowledge_search (the typed reason instead of a bare empty result set).
• A skipped extractor is not flagged. An image with no OCR/vision model skips its extractors — the declared degradation described above, reported partial. Only a node that claimed success while producing no text is the lie this names.

Memory

Stores

• vector_memory.py — semantic + episodic memory. FAISS index at ~/.personalclaw/memory.faiss (optional, the [embeddings] extra: without it semantic search compares the stored vectors directly, and without embeddings it reads by keyword), time-decay retrieval, and config-threaded episodic knobs (episodic_dedup_threshold, episodic_max_results in config/loader.py).
  • A width is only ever a vector’s. The width the model writes now is its newest stored vector’s (_width_now, the rule embedding_coverage counts “embedded” by), and consolidation reads it from the database, with or without faiss. The index holds no width until it holds a vector: a vector it cannot take (it holds none yet, or holds the model’s at another width) has it rebuilt from the database at that vector’s width, which also takes in what another store on the same database wrote meanwhile. The index holds episodes only (a fact’s or a lesson’s vector is compared row by row), so it is compared with the embedded EPISODES and never with embedded_count, which counts every memory recall compares by meaning: /api/memory/stats’ faiss_index_size counts the live episodes the index holds (a deleted one’s vector stays in it until the next build) beside episodes_embedded (EmbeddingCoverage.episodes), and the Doctor’s memory check says “N of M embedded episodes indexed” with the same numbers. Without faiss there is no index, so the stats count none, and that check, which Memory → Health shows with its Fix, says so.
  • It embeds with the model bound now. Every store — the main memory.db and each partition’s — embeds through embedding_providers/registry.py::bound_embedding(), which reads the embedding binding at each call and rebuilds when it, or the instance it embeds through, changes. A rebind, a clear or an edit in Settings → Providers reaches every store at its next use, without a restart; nothing wires an embedding function into a store (a test pins one by assigning embed_fn).
  • One model’s vectors are compared, never two. Each stored vector records the model that wrote it (embedding_model). Search, dedup, lesson dedup and consolidation compare only vectors of the model bound now, because two models’ vectors are unrelated spaces and at one width they would score numbers that mean nothing. The rest are stale, and a memory no model embedded (written while none was bound, or when it failed) is unembedded: both are read by keyword beside the vector results (search_episodic merges the two by score — a keyword hit scores the share of the query’s words it holds, weighted as a similarity is — so a memory the re-index has not reached yet stays findable, and Recall’s ranking does not put it last), counted by one reader (vector_memory.embedding_coverage: a vector of another model or of this one at another width is stale, one of neither is unembedded, and read_by_keyword is both), which /api/memory/stats, the recall disclosure, the Memory page’s Embedded stat, the Doctor’s memory.store row and the gateway’s start all read and state in one sentence (memory_ranking.keyword_read_note), and embedded by the re-index (dashboard/embedding_reindex.py), which visits every memory store, open or not. Every change of the embedding model reaches it by one path (dashboard/handlers/embedding_reindex.py::reindex_for_binding): binding Embedding (PUT /api/models/active/embedding, whoever calls it), removing the provider whose model was bound so the next in its chain is bound instead, the setup wizard’s one-click local model bind, a binding another process wrote (--seed-local-model, an edit by hand), and the gateway’s start when any store holds what the bound model has not embedded. So do another home’s memories and knowledge, which a merge brings in with the vectors their model wrote — a sync pulling a peer’s databases, a restore’s merge, an import: each says rows arrived (embedding_arrivals), and the path re-embeds what the model bound here did not embed. The gateway’s watch on the binding (watch_embedding_binding) takes the path at its start, whenever the binding changes or rows arrive (after the re-index running then, which counted what was there when it began), and again later for a model that was not ready, backing off from 30 seconds to 10 minutes, so a model bound before it could embed is re-indexed once it can. A merge run at a terminal is another process: the gateway’s next start takes it. The memory re-embed commits as it goes (reembed_stale, every 50 memories), so a stop keeps what it did, and the index file is used only when it holds exactly the vectors the database does, so a file saved before a stop is rebuilt rather than read as the new model’s. A vector written before models were recorded names none and is stale once a model is bound, by the same rule as the knowledge chunks’ fingerprint, so the first start after that update re-embeds every memory once, in the background. The index is one value (_Index: the FAISS index, its ids, their width and model), published by one assignment, and a search reads it once: a rebuild on the re-index’s thread can never hand a search its ids with another build’s index. No search rebuilds it or waits for a rebuild. A running re-index points it at its model first; each vector it writes is compared beside the index, one by one, until the re-index publishes an index that holds it (after 256 of them, and at its end, built off to the side); and a search that finds the index on another model’s vectors while a build runs on another thread reads by keyword meanwhile.
  • A recall embeds its question once, and never a stored memory. The fact and lesson ranking (_rank_rows) compares the question with each row’s stored vector: a fact is embedded when it is written (set_semantic, keyed and valued as the ranking reads it, semantic_vector_text), a lesson by write_lesson, an episode by write_episodic, and the re-index embeds whichever the bound model has not, facts included. It used to embed every fact and lesson again at every question: a store of 32 facts and 6 lessons made one recall 39 round trips to a cloud model, about 20 seconds. memory_recall’s route (GET /api/memory/recall) embeds the question once for its three arms (MemoryService.embed_query), waiting at most QUERY_EMBED_BUDGET_SECS (4 s); a model that does not answer in time leaves the recall to keyword search, and the block and the ranking disclosure say so. The route runs in a worker thread, never on the event loop (where the wait stopped every request the gateway served), and answers within RECALL_BUDGET_SECS (8 s), inside the agent’s own wait for the gateway (mcp_core.GATEWAY_READ_TIMEOUT_SECS), or answers memory_recall_timeout naming what it was still doing. Each logs one WARNING with the stage and the budget, never the question. Nothing on the event loop waits on a model: every route whose memory work can, and the putting together of every turn’s message (a chat’s, a webhook’s, a heartbeat’s, a subagent’s), hands that work to a worker thread the same way (tests/test_nothing_waits_on_a_model_on_the_event_loop.py). A recall the route stopped waiting for stops at its next stage and records nothing.
• memory_ranking.py — the ONE owner of “how did this recall actually rank”. Derives a RecallRanking from the provider’s declared MemoryCapabilities (vector / full_text_search / entity_graph) and composes the user-facing sentence server-side, so every surface presenting ranked results renders one wording instead of authoring its own. Served as ranking by /api/memory/recall, /api/memory/context-preview, /api/memory/episodic/search and /api/memory/entities; null on a recall a temporary session blocked, because no recall ran. It also reads the one piece of store state a capability cannot show: how many memories hold another embedding model’s vector (stale), how many hold none (unembedded) and how many the model bound now can compare (comparable). With none comparable the recall is keyword-ranked and says so; with some waiting it says how many were read by keyword, in keyword_read_note’s words, which the Memory page and the Doctor print too. A recall whose question the model did not embed in time carries why (question_unembedded), and reads as keyword-ranked for that reason. Adding a capability is forced to declare what it means for recall — tests/test_recall_ranking_disclosure.py censuses the dataclass fields, and a second census requires every handler calling a ranking scorer to serve the disclosure.
• memory.py — structured key/value memory with FTS5, and the markdown documents: each memory’s preferences.md and projects.md and its daily history.
  • A consolidation never undoes an edit made while it ran. It reads preferences.md and projects.md, waits for its model (seconds, for a real one) and gets back a rewrite of each, while the owner may save them in Settings → Memory or the Files editor, and the agent write them with its file tools. So a rewrite is applied to each file as it is when the model has answered (MemoryStore.rewrite, memory.apply_rewrite), line by line: each part it changes (a line rewritten, removed or added) is changed where the file still holds it as the model read it, with at least one line nobody changed between it and each change made since. A part both changed the same way is kept once. Where both changed the same lines, or lines right beside each other (a project and its notes belong together, and two additions in one place cannot be ordered), the file is left exactly as it is, that file’s rewrite is not applied in this pass, and the gateway log says so. A file nobody changed while the model ran is the rewrite, as the model wrote it.
  • Each day of the daily history is its own file, and the Memory page edits one day at a time. A consolidation appends its entry to today’s file (history/<YYYY-MM-DD>.md); the agent’s context reads the recent days together (MemoryStore.read_recent_history: the last two weeks whole, older days cut to their first entry, then to a count), and that view is never written back. The History editor in Settings → Memory lists the days (GET /api/memory/history: newest first, each with its entry count, today always among them) and holds one day’s file as it is kept (GET/PUT /api/memory/history/{day}, named by its date; any other name is refused history_day_invalid and never made into a path). Its Save replaces that day’s file and no other, through the write every memory file takes (MemoryStore._persist): refused for work that may change none of your memory, in the refusal’s own words, then written, then indexed for keyword search. Like the other documents, a save names the copy it was built from (If-Match), so an entry a consolidation appended meanwhile is not saved over.
  • One lock for every writer of the documents and the daily history. The consolidation (its rewrite of the two files and its history entry), the Memory page’s save of a document or a day, add_preference (the sandboxed agent’s memory_remember, the plain-text memory’s writes), the removal at start of what an Incognito or Temporary chat left in the history, the boot that creates the files, a partition’s documents and history moving into a project’s, and the Files editor’s save and the agent’s write_file and edit_file (write_locks.write_lock, which also holds HEARTBEAT.md’s lock for it) read and write them under memory.hold_documents, a lock file in the home’s locks/, so none lands between another’s read and its write; the personalclaw consolidate command takes it from its own process. A command the agent’s shell runs, or another program, takes no lock: it can meet a consolidation only in the instant the consolidation writes, never across its model call.
  • A writer that reaches any file writes a document as its store does. The agent’s write_file and edit_file and the Files editor’s save write a memory document through MemoryStore._persist (memory.write_document, for preferences.md, projects.md and a history file of the home’s memory or a working folder’s), so it is indexed at once and refused where every memory write is. Work that may change none of your memory changes nothing in the memory folders at all: the file tools and the shell refuse it before anyone is asked, and the sandbox keeps the folders read-only to its commands (see chat-sessions.md).
• memory_record.py — the typed MemoryRecord with a kind discriminator, the one shape the subsystem speaks. The key taxonomy is prefix-based: pref.* / project.* keys are semantic facts; lesson.* keys are corrective rules; user.procedural.* / user.persona.* / user.commitment.* are their own kinds.
• memory_service.py — the service layer, including promotion: session-scoped records are swept at session end unless sealed or promoted; promote_by_heat is the conservative global gate that promotes only records whose accumulated heat crosses the threshold (protects against one-off session noise).
• memory_vault.py — the human-readable markdown vault. memory.vault_mode picks off / mirror (projection only) / two_way (hand edits are read back through MemoryService.apply_vault_edit, i.e. the normal semantic write path with the scan and a reversible memory_events row). Every page carries a source_hash of its BODY, which is what makes an edit detectable and a frontmatter rewrite invisible; a page the parser cannot read is left alone and flagged rather than merged. Files dropped in <vault>/raw/ go to KNOWLEDGE, never into memory — the boundary holds inside the vault. Every sync (the one after a chat ends, and Sync now in Settings → Memory) is followed by the sweep (MemoryVault.sweep_raw, on the gateway’s event loop), which takes each file as the Knowledge page takes an upload (knowledge.file_items.take_file): typed by its kind (media.classify), held to the size the upload policy allows it, its bytes scanned, and kept in the library’s own files, so the reader for its kind reads it when it is ingested (a document’s text, which the ingest scans too; a script’s code, read and scanned as it is kept; a picture’s graph). A file Knowledge refuses (a kind it does not take, too large, content the scan refuses or could not check, a binary file named as code) becomes a failed item that says why and keeps no text and no file (file_metadata.refused): no step of it runs, a re-run of its ingest says why again rather than that it holds no text, and its page offers no Retry, since the file goes in again by being dropped again. Every swept file then moves to raw/.ingested/, under a name of its own when one there already has its name; an empty file stays in raw/ until it holds something, and so does a file still being copied in, which the next sync takes whole (Sync now says how many it left, raw_waiting). A note an earlier version made of a dropped file is left as it was, since the owner may have edited, tagged or filed it: to have one read the way a file is read now, delete the note and move its file from raw/.ingested/ back into raw/. Each folder’s memory that holds records is a vault of its own under folders/<id>/, in the same mode, so a hand edit there is read back into that folder’s memory (see “Partitions & project locality”).
• learn.py — lesson capture; memory_lint.py — hygiene checks; engagement_signals.py, preference_facets.py — derived preference data.

Standing instructions (brought over from other agent tools)

• What they are. The instruction files the onboarding import brings over — Claude Code’s CLAUDE.md and rules/*.md, a project’s own CLAUDE.md, Codex’s AGENTS.md — written whole, redacted, under workspace/memory/instructions/<tool>/ (onboarding_import/writers.py). standing_instructions.py carries each one into a new conversation’s session context word for word, ahead of history and recall, framed as the user’s rules. A project’s own file goes only to a conversation whose working directory is that project’s folder or below it. Temporary sessions read no memory, so they carry none.
• Which files. Only the ones the owner’s import wrote, as its ledger (onboarding/import_state.json, outside every folder an agent’s file tools reach) records them: a file something else puts in the folder is not followed. The file itself is read each time, so an edit to it is what the next conversation follows. Two files with the same text (one tool’s file linked to another’s) are carried once.
• The budget. Instructions are the first claim on memory’s share of the window (context._MEMORY_WINDOW_FRACTION, an eighth), up to 24,000 characters at any window (standing_instructions.MAX_CHARS); the memory sections scale into what they leave (context._memory_caps(reserved_chars=…)). Files go in whole, the ones that apply everywhere first. A file that does not fit is left out whole, never cut: the block names it with its path so the model can read_file it, and the turn’s notice tells the person which file, its size and the room there was.
• Memories come over as memories. The notes a tool was remembering (Claude Code’s projects/<cwd>/memory/*.md, Codex’s memories/*.md) become episodic memories in memory.db holding all of their text, split where the text breaks and numbered when a note is longer than one memory holds (vector_memory.EPISODIC_TEXT_MAX). Recall finds them, and they are embedded when written if an embedding model is bound, else by the re-index once one is. The note is kept as a file under workspace/memory/imported/<tool>/ too.

Partitions & project locality

• Memory is partitioned by working directory: config/loader.py’s memory_dir_for_cwd(cwd) maps a session’s cwd onto ~/.personalclaw/workspace/_ext/<slug(cwd)>, and an empty cwd onto the shared _ext/_default partition. context.py::ContextBuilder.get_memory_for resolves and caches one store per partition through memory_locality.partition_for, which gives the gateway’s own workspace, where every chat starts, the global partition (so a dashboard chat and the Memory UI share one main store), worked out from the path alone (memory_locality.is_the_workspace), so a process with no context builder (personalclaw consolidate) agrees with the gateway. The workspace is the default workspace each release makes, as the home or release that recorded the chat had it: this home’s (the one config.loader.resolve_workspace_root resolves, and the home’s own workspace folder), the workspace folder of a home at its default place (~/.personalclaw/workspace) on whichever machine recorded it, and the folder releases up to 0.1.3 made, ~/workplace/personalclaw-workspace. A partition gets its vector index once an embedding model is bound, asked at each use, so the first binding reaches a directory already open, or once it holds memories one wrote, so a clear leaves them searchable by keyword. A writer is handed the partition with its record store (get_memory_for(..., writes=True)), so what it keeps is kept before any embedding model is bound, read by keyword until the re-index embeds it.
• A chat’s partition is its folder’s. A chat records the folder it works in as workspace_dir, on its live session and in its transcript’s metadata (dashboard/chat_persistence.py), and memory_locality.chat_folder is the one place that is read for its memory: the turn’s own recall, the after-turn review, consolidation and its seal, and memory_recall. A project chat records there the folder its project binds, when it binds one, a loop’s worker the folder its loop binds, and a Code loop’s task worker its task’s worktree. A chat in the workspace records no folder (memory_locality.folder_on_record): the workspace is the home’s, and a path for it was true only in the home, at the place, that wrote it. A chat that records none works in the workspace of whichever home it is in, as a new chat does, and shares that home’s global memory. A restore or an archive import has each chat it brings that worked in the workspace of the home the archive came from record none (memory_locality.take_in_restored_chats): the archive’s manifest names that home’s workspace (workspaces), and an older one names the home (personalclaw_dir).
• Work done for a chat reads that chat’s memory (memory_locality.work_folder, up the chain memory_reads.reach_of walks): a subagent’s first prompt and its memory_recall read the partition of the session it works for, a workflow step’s the partition of the chat that started its run, and a project’s run its project’s, whoever started it. Each reads that folder’s partition first, then the global memory, labeled and fenced (ContextBuilder.build_message(memory_folder=…)), and searches by its task, never by the instructions the task is put under (recall_query). Only work that may read memory reads any: a Temporary chat’s and an app’s not given your memory read none.
• A consolidation marks its chat consolidated only when its answer was whole and was its own. An answer its model stopped at its output cap is no answer (the chain’s next model is asked), and nor is a JSON object holding none of the keys the pass asked for (history._consolidation_problem): a cut or unparsable answer still holds whole objects from inside it, a fact or a lesson, and read as the answer one kept nothing while the chat read consolidated. A whole answer that found nothing to keep holds the keys, empty, and does mark it. With no answer the pass keeps and marks nothing; one no model could finish is owed and tried again on the heartbeat, and said in a notice that names the output limit to raise.
• Consolidation keeps a folder chat’s memory in that folder’s partition (HistoryConsolidator._kept_in): the daily history entry and the session summary, the facts, the episodes, the persona notes, the seal, and the per-store maintenance it runs (category TTL, heat promotion, failure synthesis, daily digest, the reflex log’s retention, topology). Two things are kept in the global memory: a lesson joins the global lesson list with the folder’s reach (scope=workspace, scope_ref the folder), the list Settings → Memory → Lessons and memory_remember keep, and a proactive check-in, which the heartbeat delivers from the global memory and no prompt recalls. A chat whose folder was one of PersonalClaw’s own and is gone (memory_locality.folder_is_gone) keeps nothing more: no model call, and no partition brought back.
• A folder chat follows its partition’s lessons and, beside them, the global lesson list’s for every chat and for its folder (VectorMemoryStore.get_lessons_context, beside=): a rule taught in Settings or with memory_remember reaches it. memory_recall reads the memory of the work asking first (the work its sign-in proves, approval_answer.work_of_request: the chat its X-Session-Key names, and the global memory alone for an app’s own work, whatever chat it names), then the global memory, labeled and fenced as cross-partition recall, as that work’s prompts do.
• What an earlier version filed in the global memory for a folder chat moves to its folder’s partition at each start (memory_locality.move_what_folder_chats_left, idempotent), before anything recalls (memory_locality.settle_at_start runs it, after a chat recording another home’s workspace is made to record none, and before the naming pass, the pass below that brings back what a workspace kept, and the project pass). Each move is one line in the history of both memories (Settings → Memory → Audit): how many records moved, and to or from where. An episode is filed under the conversation it came from (consolidation and the seal set it to the chat’s key) and a session summary under its session, so each moves whole (id, text, vector, dates) to the folder the chat’s transcript names. A fact names only the last chat that stated it (every chat that learns the same thing writes the same row), and a lesson, a persona note and the daily history name none, so they stay in the global memory; so does what a chat left whose PersonalClaw folder is gone. The pass reads the chats’ metadata through the conversation log’s listing (a stat per transcript once the home’s saved listing is loaded), and removes what moved from the global memory in one go, its vector index rebuilt once; a folder whose memory cannot take its records leaves them in the global memory for the next start.
• What an earlier start moved out for a workspace comes back to the global memory at each start (memory_locality.move_back_what_workspaces_kept, idempotent): the partition of a folder that is the workspace (is_the_workspace) is the memory of no chat of its own, so everything it holds moves whole into the global memory, as a context folder’s does, and the partition is removed. Such a partition was left by a start that took a chat of a home restored at another path, or of an earlier release, for a chat in a folder of its own, or it is the memory of a folder since made the workspace. A partition is the memory of the one folder its record names, so the memory of any other folder is never taken.
• A partition’s memory_index.db holds that folder’s memories (its vector store and its full-text index share the file), so the state manifest declares it a partition of memory.db (StateEntry.partitions): a snapshot and an export copy it through the sqlite backup API, a merge restore and an archive import in merge mode merge its memories as they merge memory.db’s, and Doctor’s durability audit counts it declared.
• Because that file holds memories, nothing deletes it when the keyword index cannot be used (memory.py, “The keyword index”). The index is derived from the memory files, so an index that fails is dropped and rebuilt in place from them, and nothing else in the file is touched. A database SQLite reports damaged is moved aside whole, with the files SQLite keeps beside it, to memory_index.db.broken-<UTC instant>, a new one is started, and a notice says where the old one went. A lock, a read-only file or a full disk changes nothing: keyword search is recorded as degraded with its reason, logged once, and the Doctor’s memory.keyword-search check (also shown on Settings → Memory → Health) says so, tries the index again before it reports, and names every damaged copy still set aside.
• The learning log (learning.db: the captures waiting to be compiled, each capture pass’s outcome, the prompt budget samples, the ablation sweeps, the curator’s undo journal and the surfacing events) comes into another home row by row whichever way it arrives: a merge restore, an archive import in merge mode and a folder sync all run snapshot._merge_sqlite_attach. Its stores number its rows, and nothing else in a capture or a pass names it, so each row also has an identity, the same in every home it reaches (durability.numbered_rows.give_identity), and the merge matches rows by it: every row of the other log comes in, under a number of this home’s, and a row both logs hold comes in once, so merging the same archive again adds nothing. A row an earlier version wrote gets its identity when the log is opened, made from its number and what it holds, so the same row has the same identity in every home that holds it and in an archive taken before. The merge opens both logs as their stores open them first (StateEntry.open_database, learning.staging.open_log), so a table of the other log that this home’s stores have not made yet is made and takes its rows. A proposal names the captures it was compiled from by their identities (staging_refs, StagingStore.sources_for), and the curator’s journal reads its changes newest first by when each was made, not by number. The evidence a lesson’s confidence stands on (lesson_evidence) is kept by lesson and never added up by a merge.
• A partition’s learning.db holds the evidence its lessons stand on (VectorMemoryStore._lesson_evidence_store; a lesson with none falls below the confidence gate), so it is declared a partition of learning.db and travels and merges the same way, by a merge restore and an archive import alike. It stays the partition’s: a global lesson has the same key in every partition, and one shared file would let a reversal in one void it in all.
• A folder of PersonalClaw’s own that sessions run in, a task’s git worktree (loop/worktree.py::SESSION_FOLDERS) or a loop’s folder, where a planner with no workspace works (loop/files.py::SESSION_FOLDERS), takes its partition with it: memory_locality.drop_partition runs where the folder is removed (remove_worktree, a project’s or a loop’s delete) and closes the stores held open on it first, and memory_locality.settle_partitions removes, once at the start, the partition of such a folder that is gone. It reads a folder back out of a partition’s name only when that name is exactly the folder’s partition name; one shortened to a hash is left.
• Every file of a partition is 0600 and every folder 0700 from its first byte: its databases open through sqlite_compat.connect/connect_shared, which make the file private before SQLite first opens it (atomic_write.make_private_database), and the start-up pass makes a partition an earlier version left readable private too.
• Every folder’s memory is listed, named by its folder. A partition records the folder it is the memory of in its folder.json the first time a store is opened on it (memory_locality.record_folder), and the record is believed only when that folder’s partition name is the partition’s own (recorded_folder), since a slug cannot be read back into a path. A start-up pass names the partitions an earlier version left unnamed from the chats’ folders and the projects’ (memory_locality.name_partitions). memory_locality.partitions() lists the global memory first, then each folder’s; GET /api/memory/partitions adds what each holds, whether its folder is still there, and the projects it is the memory of.
• Settings → Memory shows and manages whichever memory you pick (“Memory of”, shown once there is more than the global memory): every route the page reads and changes memory through takes ?partition=<id> (handlers/memory.py::asked_partition), so the facts, lessons, episodes, documents, graph, recall test, audit, health and export it shows are that memory’s. An id that names no memory is 404 memory_partition_not_found, never read as the global memory, and an id is a partition directory’s name exactly (letters, digits, ., _, -; never ., .., the global partition’s or a link). What is the global memory’s alone stays so: the vault’s status and sync, migrating legacy markdown, consolidation, the triage approval rules and the memory settings.
• A folder’s memory can be removed (DELETE /api/memory/partitions/{id}, the picker’s “Remove this memory”, confirmed): the stores held open on it are closed and its directory removed (memory_locality.remove_partition), with a security-log row. One whose folder is gone is listed as such, so it can be found and removed; the global memory has no id to be removed by.
• memory_list and memory_forget read and change the lessons of every memory (/api/lessons?partition=*), each listed lesson naming the folder whose memory keeps it, and memory_forget saying whose it removed them from. A lesson is added to one memory: POST /api/lessons?partition=* is refused. On the command line, personalclaw learn list and remove reach every memory the same way, personalclaw memory partitions lists them with their ids, and --partition <id> points memory list, search, stats, export and import at a folder’s (memory_locality.record_stores).
• The vault mirrors each folder’s memory as a vault of its own under folders/<id>/, linked from the root MEMORY.md under “Folders” (memory_vault.MemoryVault, folder=). A folder vault whose memory was removed is retired, only the files its manifest lists, and the root vault’s lint never reads a folder vault’s pages as its own orphans.
• One project keeps one memory: the partition of the folder it binds (memory_locality.project_folder), or the global memory when it binds none. Its chats work in that folder and keep their memory there, so its runs read that memory too, whatever folder their steps work in (its context folder, a scratch folder or a worktree, none of which keeps memory), and so does its routed context (get_context and the context files: the project’s memory first, then the global memory’s episodes, each said to come from outside it). The project’s context folder would split the bound folder’s memory between the project’s chats and that folder’s other chats, and take an unbound project’s chats away from what they have kept. What an earlier version’s runs kept in the context folder’s partition moves to the project’s memory at the start (memory_locality.move_what_context_folders_kept, idempotent): facts and episodes record by record (of two facts under one key, the newer stays), lessons with the evidence they stand on, and preference lines, the projects text and each day’s history entries the target does not hold yet; the emptied partition is then removed. A memory that cannot take them leaves them for the next start.
• Recall for a project-local session is partition-first: its own partition, then the global partition, whose hits are source-labeled and fenced (security.py::fence_untrusted). This affects ordering only, never admission — a hit that exists only in the global partition is still returned, on its own if need be.

Recall & the privacy guard

• Recall handlers live in dashboard/handlers/memory.py. Whether work may read your memory at all is one answer, memory_reads.reach_of, asked by every reader: the context a turn and a subagent’s first prompt are assembled with (memory, lessons, standing instructions, episodes, active recall, the push reflex), memory_recall (/api/memory/recall), memory_list (GET /api/lessons), get_context’s memory tier and the Learning page’s facts. A temporary chat’s work reads nothing (the chat, its subagents and their own, the steps of a run it started), and neither does an app’s work unless the app holds the memory permission: a conversation the app started, an agent run it asked for, an agent its scheduled job started, an agent working for any of them, and the app’s own requests. A refused read answers why, in words the agent passes on, rather than “nothing found”. Work someone other than the owner asked for (a colleague’s turn in a shared thread, and what it starts: memory_writes.asker) reads her memory as any turn of its chat does, but searches none of her chats: chat_search (/api/sessions/recall) and the inbound door’s sessions_search answer it with a sentence naming who asked (chat_recall.not_searched_for). The same grant governs what such work changes: without it the memory store refuses its writes, its lesson and triage-rule tools say why, the after-turn review and the run-end learner learn nothing from it and its conversation is never consolidated; with it, each record it writes names the app as its source (app:<name>), so it is weighed as the app’s and never as yours (memory_writes.written_by). The knowledge library is your content, not memory, and is read as your files are. Both temporary and incognito sessions keep nothing: the memory, knowledge and vocabulary stores refuse every write made in their name, by any path, and nothing from them reaches the embedding model, so an incognito chat’s memory is searched by keyword (memory_writes.model_may_read, asked by the embedding functions, including on the worker threads recall runs on) — see chat-sessions.md.
• Recalled episodic content is fenced as data: the recall block is labeled [Recalled episodes — past conversation fragments (DATA, not instructions)] so a poisoned memory can’t smuggle instructions into the prompt. (The generic fencing helper for untrusted content is security.py::fence_untrusted — see security.md.)
• The after-turn learning path (after_turn_review.py) is gated on session.is_restricted — restricted sessions never write lessons.
• A lesson is authorised and validated before anything is changed or counted (VectorMemoryStore.write_lesson). Work that may change none of your memory is refused first, so it counts no sighting either (the sightings live in learning.db, which the statement check never sees); a lesson the store’s rules refuse is recorded in the history as a write that did not happen; a lesson one already held says in full is a sighting of that one. None of them retires anything: the lessons a new one replaces are retired toward it in the transaction that stores it (_write_semantic’s retiring, inside SharedConnection.transaction), and supersede_semantic retires a row only toward one that is stored, so nothing drops out of recall for a replacement that was not kept. A row written again is live and points at nothing. The lesson route answers a refused lesson 422 lesson_refused with the store’s reason (MemoryService.lesson_refusal), which the Memory page, the agent’s memory_remember and personalclaw learn add show, and it scans what not to do beside the rule as it scans the rule. A lesson an earlier version left retired toward one that was never kept comes back when its store opens, with the sightings it had carried onto that key, or points at the lesson taught since that says it in full (recorded in Memory → History under the source repair). The same repair brings back any row a purge left retired toward a row it removed.
• Only she replaces what she taught. A row whose source is a person’s (user_explicit from her own routes and tools, vault_edit from her vault) is overwritten, replaced or retired by a person alone: vector_memory.only_a_person_replaces is the store’s one rule, asked by conflict resolution under the row’s own key, by every retirement toward another row (_retire_toward, so supersede_semantic too), by the lesson dedup and by consolidation’s formation. A lesson something else learned (a chat’s consolidation, the after-turn review’s correction or veto, an app’s work, a migration) that would replace one of hers, through the dedup or through the contradiction judge’s verdict (asked about her lesson before anything is kept, and never counted against it), keeps nothing and retires nothing: hers stays as it is and she is asked which to keep (learning.lesson_conflicts: a lesson_batch proposal tagged lesson_conflict, on the Learning page and in the Inbox, naming both rules and the memory hers is in). Accept saves the learned lesson as hers in place of hers (VectorMemoryStore.keep_lesson_in_place_of, undoable in Settings → Memory → Audit); Reject keeps hers, and the queue’s decision memory never asks the same question again. Her own new lesson replaces her older one as before. Formation’s supersession or cross-key update of a fact she set keeps both, flagged as a conflict, and its extract-phase removal of one is not applied.
• Two lessons are one rule said again, and the newer replaces the older, when the significant words they share number at least two and are at least half of all the words either holds (_RESTATED_SHARE, _RESTATED_WORDS), or one contains the other, or their vectors are more alike than 0.85. The share was measured over the smaller lesson’s words alone, so one shared word retired a rule of two words; and the framing every correction lesson opens with counted as shared words, so any two short corrections were one rule. That framing is no word of a rule now.
• Deleting a chat forgets what memory drew from it alone (chat_forget.delete_chats, through memory_writes.forget_what_sessions_left, the one purge the start-up sweep of Incognito and Temporary chats runs too). A record is the chat’s when it is filed under it (source_session): its episodes and its sealed summary, its running summary, and every fact, lesson, persona note, check-in or tool-outcome record its own work wrote (its consolidation, its turns’ memory tools, its after-turn review, the subagents and runs working for it) that no other work wrote or confirmed since. They go from the global memory and every folder’s, with their history events, links, vectors and the names they alone mentioned among the graph’s proposals, the daily-history entries that repeat any value they held, and the days’ digests that quoted their episodes, built again from the episodes left; a memory’s vault, when one is kept, is written again without their pages. What other work also stands behind is filed under no one session (vector_memory.SHARED) and stays: a lesson said again in another chat, a fact rewritten from the Memory page or the command line, a fact the formation pass judged already kept, an episode told twice, and what consolidation’s maintenance writes over every chat (memory_writes.as_maintenance; an earlier version filed that under the chat it followed, and the store’s migration v13 refiles it under none). The Delete dialog says what goes and what stays.
• Learning reads only what the person typed. Every per-turn capture (the correction lesson, preference facets and vetoes, the glossary slot, the self-model observer, stumble refinement and the skill ladder) is handed own_words.own_words(row) of the row that started the turn, never the message the model was sent: that message also holds a saved prompt’s text in place of its @name, an attached file’s or a referenced item’s text, a theme’s persona and natural voice’s instructions, and a turn an automation, a subagent’s report or a heartbeat started holds no words of hers at all. A pasted block and a fenced span are left out too. The code that composes a row records which of its words were typed (meta.own_words: the send, a queued or merged send, plan mode’s prompts, a heartbeat’s delivery), and a saved prompt’s expansion records meta.ran_prompt {name, text}, which the chat shows folded under her message as the prompt’s. A send cannot set either. Consolidation reads a person’s saved rows the same way (history.consolidation_line): what they typed as theirs, each pasted block and a row sent in their turn as material that is not, and a prompt they ran by its name; a promoted skill’s and a project review’s proposals quote only what they typed.
• Learning reads only the owner’s words. A conversation on a chat channel can have other people in it, and the door lets in everyone the owner trusts to talk to the agent: a friend in a group, a colleague in a shared thread, a correspondent writing to the mailbox, anyone an open direct message admits. What they write enters the chat as a user line, unfenced, because the agent may act on what they ask, and none of it is the owner’s. own_words.own_words(row) reads a row only when the owner sent it (turn_source.sent_by_owner): the dashboard took it in, or the row names its channel (source_channel) and its sender (source_user) is the owner that channel keeps (owner_id_for, the id its owner pairing, or its setup, stored; Slack’s two spellings of one member both match). On a channel that knows no owner nobody is the owner, and any other source is nobody’s too: a program through the OpenAI-compatible door, a channel’s row saved before rows named their channel, one saved with no sender. So a turn someone else started teaches no lesson, preference, veto or glossary line; a row several queued messages run as keeps only the words the owner sent (own_words.queued_words); and consolidation is shown nothing of a turn someone else asked for (own_words.her_turns): not their message, not what the agent did or said for them, only what she sent in it (her part of a row her queued message and theirs run as, a message she sent into it while it ran). A row belongs to the turn the row that started it began (turn_source.by_turn), so the agent’s answer to them is theirs too. No label stands between their words and her memory, since a model can read past a label: what holds is that the model is never shown them. A row that records no source at all (an imported conversation) reads as before. What you send on a channel counts as yours once the channel knows you as its owner: its Configure page says who that is.
• Memory changes only on the owner’s word. A turn someone else asked for (the row that started it, by the same rule: turn_source.asked_by; a row several queued messages run as is asked for by everyone who sent one, which it records under meta.queued_from) names who asked once its row is known (memory_writes.asked_for, said by the turn engine), and a channel that runs a conversation itself names it around each turn (turn_asked_by, the turns-name-who-asked core feature). memory_writes.asker is then the answer for the turn’s own work, for every request its tools make while it runs (which name its session, read from the registry the turn marks), and for a subagent it starts (hand_on marks it). Such work writes nothing to her memory: the stores refuse its changes (refuse_memory_write, the memory database’s statement check), saying who asked, under the code asked_by_someone_else, and so do the file tools, the shell, and the gate an agent CLI asks before one of its own tools runs, for a memory document (see chat-sessions.md); a read leaves no mark, and the turn’s own learning takes nothing of what the turn did: not its tools’ outcomes, which procedural memory, the self-model observer and stumble refinement read (they come off the agent unread, so the next turn is not credited with them), nor the answer the skill ladder drafts from. What that learning reads of the turn’s message is the owner’s words alone (own_words), so it is asked for by whoever sent those words (turn_source.taught_by, run under memory_writes.learning_from_words): her message, queued and run beside someone else’s, still teaches as hers, her correction a lesson and her preference a facet, while theirs teaches nothing (GateReason.ASKED_BY_SOMEONE_ELSE), as consolidation reads it. What the agent’s memory tools ask for in such a turn is held for her instead (dashboard/memory_holds): memory_remember, memory_forget and triage_rules answer 202 with a sentence the agent reads (nothing written, who asked, the owner was asked), and she is asked once through the approval registry (workflows.owner_allow), wherever an approval reaches her: the card, the Inbox, the phone, the channel the chat is on, and her “Send approvals to” channel when the Approval needed rule sends there. No standing grant or Trust answers it. Her Allow writes the change as hers (memory_writes.on_the_owners_word: a lesson is user_explicit); a Deny, an ask nobody answered in her approval window and one whose turn was stopped write nothing, and each hold and each answer is a security-log row under the route’s operation. Her Deny of a change to her memory holds later too, in any turn: a memory tool’s call (mcp_memory.CHANGES_MEMORY), or a file tool’s or a command’s change in the memory folders, read from the approval row the call wrote (declined_calls.declined_a_memory_change). That turn teaches nothing (GateReason.MEMORY_REFUSED), and consolidation is shown nothing of it, so neither its episodes, the sealed summary, preferences.md nor the daily history holds what she refused to save. A skill drafted from such a turn (skill_remember) is refused, the agent told to propose it with skill_promote, which the owner accepts in the review queue; a decision resolved in it says why it kept no lesson. Consolidation is PersonalClaw’s own pass over the conversation, shown only the turns the owner asked for (own_words.her_turns), so it runs whoever asked for the turn it follows: its gate is asked as the session’s own pass, never as that turn’s work (chat_utils._maybe_consolidate). A pass with no turn of hers to read asks no model and keeps nothing; its messages count as consolidated, and the memory’s maintenance runs after it as after any pass. A session of lasting work someone else asked for (a loop’s, a run’s step, a callback’s turn: lasting_work.asker_of) keeps nothing at all (HistoryConsolidator.why_nothing_is_kept).
• Work such a turn starts that outlives it keeps who asked, for as long as it lasts (lasting_work.py). A workflow run, a loop and a callback record the source of whoever asked on their own record when they are made (lasting_work.ASKED_BY: a run’s extra, written where every run is made, workflows.store.create, and carried by a sub-run and a fork, ownership.inherited_extra; a loop’s row, loop.store.create; a callback’s registration, webhook_callbacks.register), and a batch keeps it while it waits for her Allow (workflows.batch_start). memory_writes.asker reads that record (lasting_work.asker_of) for each session along the chain of work it walks, after the turn has ended and after a restart: a run’s steps and its own work (which runs as its own, run_start.run_context), a loop’s workers and planner, a callback’s turn, and the run-end and loop-end learners. A turn of such work whose message is the owner’s, or that has none (a loop’s cycle, started by its nudge), runs as asked for by whoever the record names (memory_writes.asked_for), and so does the turn that hands a subagent’s report back to its chat, which reads the subagent’s mark (memory_writes.asked_for_work; a report that waits behind a turn carries it in the queue), and the report’s row records it (turn_source.ASKED_FOR_BY), as a message’s row records who sent it: a report handed to an idle chat gets its row too, so the turn it starts reads as theirs. What any of it would change of her memory is held for her word, or refused where nobody can be asked (an Unattended loop’s cycle). A record that cannot be read names someone no record names (turn_source.UNNAMED), so its work is held as someone else’s. The tool server an agent CLI runs holds no turn, so it asks the gateway who asked (GET /api/chat/sessions/model-reach, its asked_by). An automation is not made or changed on someone else’s say-so at all, and a loop is steered only with the words of whoever asked for it: a tool says so in its error and a route answers 403 asked_by_someone_else, as a private chat’s work is refused them.
• Every history a model is handed of a conversation names who said each line, by the same rule (turn_source.turn_line): the owner’s line as User:, anyone else’s whole and fenced as SENT BY SOMEONE OTHER THAN THE USER (not the user's words) (a merged row that holds both as SENT BY THE USER AND SOMEONE ELSE TOGETHER). That is the turns a fresh runtime is given back after a restart (prior_turns_transcript, history.model_view, history_for_model), the compressed history (compress_thread_history), the summary background compression keeps for an idle chat (bg_compress), and a stopped turn handed back to the next one; each row keeps what it records of where it came from on the way (turn_source.provenance), as ConversationLog.recent does for what a promoted skill’s and a project review’s excerpts quote. The turn itself reaches a live runtime as the request it is, so an agent CLI that resumes its own session, and the native runtime’s own compaction, keep a colleague’s earlier request as it was sent.
• What learning took from such text before is settled at each gateway start (learning/composed_text.settle, over every memory store in the home): a correction lesson that quoted the platform’s own opening is retracted through the memory log, so Memory → History can undo it, and a lesson it had displaced is restored; a veto, preference or glossary line that matches a saved prompt’s or a theme’s text is offered as one proposal (Learning → Proposals lists each item and the text it matches, and the Inbox notes it), whose Accept removes them and whose Reject keeps them for good.

Lexicon

lexicon/ — user terms + learned corrections in lexicon.db. The lexicon biases ALL speech transcription, within a hard budget (~64 terms / 200 chars — Whisper’s initial-prompt window is 224 tokens; overflowing it silently empties transcripts). Graph resync prunes stale terms while preserving user-pruned flags.

Its graph terms follow the knowledge graph by themselves: every consult (the bias hook, the correction node, the microphone route, the Vocabulary list) goes through current_lexicon(), which resyncs whenever the graph’s fingerprint (entity count, latest change, surface lengths, mention count) differs from the one stored at the last sync, in one transaction. Rebuild forces the same sync. Ranking decides whose names fit the 200 characters: graph_weight puts people first, then named things, then concepts, and within a type the most-mentioned; every graph weight stays below a manual term’s 2.0, and a resync never lowers a weight a learned correction raised. A graph term cannot be deleted (it would return at the next sync); turning it off sticks.

Related docs

• Which model runs each pipeline stage: bindings in overview.md
• Event triggers that fire on memory writes: tasks-triggers.md
• The egress policy connectors fetch under: security.md