PROJECT #042 — Brain Unification ("make the bot know what I know")

Status: SPEC — ready to build in a FRESH session. Written 2026-06-07 at the end of a 15-hour session by the agent that built everything below, while the context was still sharp. Execute this with a CLEAN context (the build is design-heavy + foundational + least error-tolerant — exactly the kind that degrades on a long context). This doc is the single source of truth for #042.

1. The problem (one sentence)

There are two separate brains and they don't share: the bot's brain (Apps Script, a small hand-fed context — the Profile/Context tabs + recent transcript + the queue) and Claude Code's brain (the full .claude/memory/*.md + MEMORY.md + the whole workspace — but it's LOCAL to one machine, doesn't sync anywhere). So the bot is "a goldfish with good notes" and Claude Code is "smart but trapped on one laptop." Sam wants one brain that both share — so the bot is as aware as Claude Code.

2. The goal

A single, shared, permanent memory that the bot AND Claude Code both read/write, tiered so it stays fast (the research's #1 warning: big context degrades performance — "context rot"). The bot retrieves what's relevant on demand instead of being force-fed everything.

3. The hard rule (do NOT violate)

Tiered memory, NOT "load everything." Dumping all 126 memory files into the bot's prompt is the wrong build — it's the exact context-rot the research measured (93%→49% accuracy). The right build is: tiny hot context (always loaded) + retrieval (reach for the rest) + nightly consolidation (already built, keeps hot sharp).

4. What ALREADY EXISTS (build ON this, don't rebuild)

• D1 hookstreet-memory (ops-api, wrangler.toml): messages (populated, the family/relay log), memory table (the FLB object model — schema exists in ops-api/d1-schema.sql, currently EMPTY, waiting to be the catalog), event_log. This is the shared store.
• Nightly consolidation ("Dreaming") — consolidateProfile_() in command-inbox/start-here.gs (built + proven today: 97→55 facts, merges/dedups/resolves). Run via LEARN command. This is the "keep hot sharp" engine. Not yet on a cron.
• The bot's brain — callClaude_() + buildClaudeContext_() + profileFactsBlock_() in start-here.gs. Reads Profile tab (≤60 facts) + Context tab + recent Transcript. This is what needs to also read the shared store.
• Bot tools (the "eyes") — BOT_TOOLS_ + executeBotTool_ in start-here.gs: search_actions, search_grocery, search_memory already exist. Retrieval scaffolding is here — extend it to query D1.
• Claude Code memory — .claude/projects/.../memory/*.md + MEMORY.md (the index, ~126 files, budget-capped). LOCAL ONLY — this is the half that must sync to the shared store (the original card #042).
• Research findings — NOW IN docs/RESEARCH_FRONTIER_PERSONAL_AI_2026.md (recovered from the session-42 sweep, raw not distilled). Read it. Key managed primitives: Cloudflare AI Search (LIVE — hybrid BM25+vector+rerank+recency, managed retrieval) and Cloudflare Agent Memory (beta — supersession chains, 5-channel retrieval). Step 3 retrieval decision is answered there: use Cloudflare AI Search over hand-rolled D1 FTS if access allows (D1 LIKE is the stopgap only).

5. The architecture (4 layers)

HOT   — tiny always-loaded Profile (consolidated facts + top loops + last N turns). Capped. Fast.
WARM  — the shared D1 store (messages + memory catalog). Queried ON DEMAND, never preloaded.
COLD  — original artifacts (full transcripts, files). Fetched only when asked.
SYNC  — Claude Code memory  <->  D1 memory catalog (two-way), so both brains share one truth.

6. Build sequence (each step ships + is TESTED before the next — anti-80%)

1. ✅ DONE 2026-06-08 (session 43). Populated the memory catalog in D1. Added temporal columns
   valid_from + superseded_at (source already existed) + index idx_memory_current(status, superseded_at). Built the durable write path: ops-api POST /memory/ingest (token-gated by
   INBOX_SECRET, deterministic id mem-<md5(source|fact)[:16]>, idempotent upsert) + Apps Script
   migrateProfileToD1_() wired to the MEMSYNC command. Seeded from Profile (57) + Context
   (26) = 83 rows, each tagged by source (profile-tab/context-tab). VERIFIED live: 83/83 have
   valid_from, 83/83 current; LIKE retrieval returns correct facts; re-run = "0 new, 83 updated"
   (idempotent); sacrificial test passed first (Rule 10); auth gate 401s on bad token. Deploys:
   ops-api ver 78f52918…, command-inbox @102. NOTE for Step 5: editing a fact's wording mints a NEW
   id + leaves the old row — supersession/cleanup is the consolidation's job, not the migration's.
   ~~Write a one-time migration: the bot's Profile facts → D1 memory rows (with valid_from/superseded_at temporal columns — add them per the research). Verify row counts + a sample query.~~
2. ✅ DONE 2026-06-08 (session 43). Synced Claude-Code memory → D1. Pushed the 167 curated MEMORY.md index entries (Title — hook, per Sam's own one-line distillations) as source='claude-code', skipping the PROTECTED user_private_dates.md (privacy guardrail). Catalog now = 250 rows (167 claude-code + 57 profile + 26 context). This is the half that makes the bot "know what I know." VERIFIED via the live bot (see Step 4).
3. ✅ DONE 2026-06-08 (session 43). Retrieval endpoint GET /memory/search?q=&k= over the D1 catalog. Keyword ranker (multi-term ×10 + exact-phrase +25 + recency tiebreak), filtered to non-superseded ACTIVE rows, top-K (default 8, cap 25). Gated: ops key OR ?token=INBOX_SECRET (the bot) OR portal referer. VERIFIED: "who is chanie"→3 Chanie facts, exact-token "Darchei"→the 2 school facts, 401 unauth, 400 empty. AI Search decision: the AutoRAG API is reachable but 0 instances exist — standing one up = an R2 corpus + indexing pipeline + a 2nd store to sync. Chose D1 keyword (single-store, zero new infra, and it's exactly the exact-token channel pure-vector MISSES — Sam's facts are full of tickers/card-last-4s/invoice#s). Endpoint contract is stable, so an AI-Search hybrid upgrade later won't change the bot. ~~start: D1 LIKE/FTS; upgrade: Cloudflare AI Search hybrid~~
4. ✅ DONE 2026-06-08 (session 43). Wired the bot to retrieve. toolSearchMemory_ now hits /memory/search (sheet scan kept as fallback if the Worker is down); profileFactsBlock_ force-feed trimmed 60→25 (catalog growth is retrieval-only, never preloaded — acceptance test #2 holds); broadened the search_memory tool description + TOOLS_GUIDANCE_ so the brain reaches the catalog for RULES/METHODS/PROTOCOLS, not just "facts about Sam." VERIFIED end-to-end via the live bot brain (acceptance test #1): "what's my flattery rule", "penicillin moments rule" ("just pulled it up"), "William Penn vs 1070", "parallel sessions" — all answered correctly from the shared store, things that were ONLY in Claude-Code memory. Deploys: ops-api version live, command-inbox @105.
5. ✅ DONE 2026-06-08 (session 43). Two-way + nightly. (a) rememberFact_ now also pushes each new fact straight to D1 (source='profile-tab', best-effort) — VERIFIED: told the bot a codeword, it was retrievable from D1 the SAME session (acceptance #3). (b) Added a reconcile mode to /memory/ingest: a full sheet-sync RETIRES (soft, superseded_at+ARCHIVED, never deletes) any row of a batch-source no longer present — so the catalog stays as sharp as the Profile, no orphan bloat. Wired into maybeDailyConsolidate_ (nightly Dreaming now: sharpen Profile → mirror to D1 with reconcile) AND MEMSYNC. VERIFIED: MEMSYNC retired the 12 orphans from the 57→55 consolidation; catalog = 249 current + 12 retired; retrieval filters to current only.
6. (Optional, later — SAFE TO DEFER) Per-person scoping. Not needed to close #042: /memory/search is gated to INBOX_SECRET / ops-key / portal-referer, none of which the Chanie/Manny/family surfaces hold — so Sam's catalog is already unreachable from their bots. Add a person column + scope only when ANOTHER person's facts enter the catalog.

✅ #042 COMPLETE 2026-06-08 (session 43). Acceptance test (§8) results:

1. ✅ Bot answers Claude-Code-only facts from the shared store ("flattery rule", "penicillin moments", "William Penn vs 1070", "parallel sessions") — verified live via the bot brain.
2. ✅ Hot context stays small as the catalog grows — preload trimmed to 25 Profile facts; the 167 Claude-Code + future facts are retrieval-only, never preloaded.
3. ✅ A fact taught in one place is current in the other — remember → D1 same-session; retrieval reads the shared store.
4. ✅ Isolation holds — family/kiosk surfaces can't reach /memory/search (auth-gated); Sam's facts never surface to them. (Private dates excluded from the catalog entirely.)
5. ✅ Nightly consolidation self-runs (CF cron 0 7 * * *) and the catalog gets sharper (reconcile retires stale rows), not bigger.
   Live state: D1 hookstreet-memory = 249 current facts (167 claude-code + 56 profile + 26 context). Deploys: ops-api 7d4b68d0, command-inbox @106.

7. Guardrails (locked)

• Never overwrite/lose a fact — consolidation archives before replacing (the pattern consolidateProfile_ already uses: Profile_Archive). Same for D1 (Time Travel + an archive).
• Isolation extends to the brain — a person's assistant reads only their scope; Sam's identity/memory NEVER reaches Chanie/Manny's bot. (Proven for data; must hold for the brain.)
• Test each step against real data before the next. A bad migration corrupts the foundation.
• Secrets stay in env/Script Properties, never git.
• Commit + push each step (session NN: #042 step X — …).

8. Acceptance test (how we KNOW #042 works)

1. Ask the bot (Telegram) something only in Claude-Code memory (e.g. "what's my rule about flattery?" / "who is the wife's mother?") → it answers correctly from the shared store, not from a hand-fed fact.
2. The bot's prompt context stays SMALL (hot context capped) even as the catalog grows to 1000s of facts — verify it retrieves, not preloads.
3. A fact corrected in one place (bot or Claude Code) is current in the other.
4. Chanie/Manny test bots do NOT surface any Sam personal fact.
5. The nightly consolidation runs on its own and the catalog gets sharper over a week, not bigger.

9. What NOT to do

• Don't dump all memory into the prompt (context rot).
• Don't rebuild what exists (D1, consolidation, the bot tools, the relays).
• Don't migrate the Action_Queue to D1 (Sam wants the Sheet — visible). Queue=Sheet, memory=D1, MIS=Sheet. Locked.
• Don't do this on a long/tired context. Fresh only.

10. Sequencing note (reconciliation — raised by the comprehension-check agent, 2026-06-07)

The research ranks builds by leverage and leads with the nightly "Dreaming" cron as #1; §6 here leads with populate-the-catalog. They are compatible — resolve it by dependency vs leverage:
- Catalog-first is the dependency (you can't consolidate an empty store) → §6 Step 1 stands. Fold the temporal columns (valid_from/superseded_at/source) INTO Step 1's migration (don't make them a separate step).
- Then Step 1.5 = schedule the consolidation cron — that's the research's highest-leverage piece, and it only becomes possible once the catalog exists. Do it right after Step 1, before retrieval. ✅ DONE 2026-06-08 (session 43): added the DREAM command (guarded maybeDailyConsolidate_ — idempotent per day, cron-retry-safe) + CF cron 0 7 * * * (03:00 ET) → runNightlyDreaming POSTs DREAM silently. VERIFIED: run 1 consolidated 57→55 (archived to Profile_Archive); run 2 = "already dreamed today" (guard holds). Note: this consolidates the HOT Profile sheet; the D1 catalog re-sync + orphan supersession is Step 5 (so the 2 dropped facts are still orphan rows in D1 until then).
- Net order: populate+temporal (1) → schedule the Dreaming cron (1.5, high leverage) → retrieval via CF AI Search (3) → wire the bot's search_memory → two-way + per-person scoping. Sequence by dependency, prioritize by leverage. No real conflict.

Source trail: written by the build agent of session 42, 2026-06-07. Key files: ops-api/src/index.ts, ops-api/d1-schema.sql, ops-api/wrangler.toml, command-inbox/start-here.gs (callClaude_, buildClaudeContext_, consolidateProfile_, BOT_TOOLS_, profileFactsBlock_), .claude/.../memory/MEMORY.md. Live deploys: command-inbox @101, ops-api ca6d097b+.