跳到主要内容

Memory Trees

The Memory Tree is OpenHuman's knowledge base. It is not a vector database with a thin "memory" wrapper. It is a deterministic, bucket-sealed pipeline that turns the messy stream of your day - chats, emails, documents, integration sync results - into structured, queryable, summary-backed Markdown that lives on your machine.

What it does​

Every source you connect feeds the same pipeline:

source adapters (chat / email / document)
        |
        v
canonicalize    normalised Markdown + provenance metadata
        |
        v
chunker         deterministic IDs, <=3k-token bounded segments
        |
        v
content_store   atomic .md files on disk (body + tags)
        |
        v
store           persistence (chunks, scores, summaries, jobs)
        |
        v
score           signals + embeddings + entity extraction
        |
        v
source / topic / global trees   per-scope summary trees
        |
        v
retrieval       search / drill_down / topic / global / fetch

The hot path (canonicalize → chunk → fast-score → persist → enqueue follow-up work) is fast. Heavy work - embeddings, entity extraction, sealing summary buckets, daily digests - runs in background workers so the UI never blocks.

Embeddings and summary-tree building can run on-device via Ollama if you turn on Local AI; otherwise they go through the OpenHuman backend like any other model call.

Three trees, three scopes​

  • Source trees, per-source rolling buffer (L0) that seals into L1 → L2 → … as it fills. One per Gmail label, one per Slack channel, one per uploaded document, etc.
  • Topic trees, per-entity summaries materialized lazily by hotness. The more an entity (person, project, ticker, repo) shows up, the more aggressively its topic tree is built and refreshed.
  • Global tree, one daily global digest across everything ingested that day.

Retrieval can target any scope: search a single source, drill down a topic, or pull the global digest.

Where it lives on disk​

Inside your workspace (default ~/.openhuman, or whatever OPENHUMAN_WORKSPACE points at):

PathWhat's there
memory_tree/chunks.dbChunks, scores, summaries, entity index, jobs, hotness
wiki/The Markdown vault - see Obsidian Wiki

Everything is local. Nothing about your raw data leaves your machine unless you explicitly send a chat message that includes it.

Why a tree, not a vector store​

Vector stores answer "what is similar to this query?" Memory needs to answer more than that:

  • What happened today? (global digest)
  • What's the latest on this person? (topic tree, hotness-driven)
  • What did the Stripe webhook say last Tuesday at 3pm? (source tree + provenance)

Trees give you compression and navigation. Embeddings still live inside so semantic search keeps working, but the structure on top is what makes the memory feel like a brain instead of a bag of fragments.

How the pipeline works​

1. Ingest​

A new chat / email / document arrives. The hot path canonicalizes it into Markdown, splits it into bounded chunks with deterministic IDs, runs a cheap fast-score, persists everything in a single transaction, marks each chunk as pending_extraction, and enqueues follow-up work for the workers.

2. Queue​

Follow-up work lands in a durable job queue:

KindWhat it does
extract_chunkDeep score + entity extraction. Decides admitted vs dropped.
append_bufferAdds an admitted leaf to the source (or topic) tree's L0 buffer. May trigger a seal.
sealCompresses an L0 buffer into an L1 summary; cascades up if the parent buffer is now full.
topic_routeRoutes a leaf into per-entity topic trees, gated by a hotness check.
digest_dailyBuilds the global daily digest node.
flush_staleForce-seals buffers that have been sitting too long.

3. Workers​

A small pool of background workers (3 by default) picks jobs off the queue and runs them.

4. Tree state​

Three independent trees are built from the same leaf stream:

  • Source tree - one per source
  • Topic tree - one per high-hotness entity
  • Global tree - one tree, growing one node per UTC day

See also​