{"repo": "arjundivecha/personal-knowledge-system", "files": {"README.md": {"4b57eff3dc3a9f07e5fe41e4379579f96a231699": {"t": 1786068122.0950994, "n": 34203, "h": "2800dac9837c2523", "c": "# Personal Knowledge System\n\nThis repository is building a personal memory layer for AI assistants.\n\nThe goal is not just to store past conversations. The goal is to turn a long stream of chats, coding sessions, repositories, and email into a living memory system that can:\n\n- remember stable identity and project context\n- surface the right memories for the current question\n- avoid flooding the model with one-off facts\n- gradually consolidate what matters\n- eventually \"dream\" over old material and keep only the useful residue\n\nThis README is meant to explain the system, not to present it as a turnkey template. The codebase is private, opinionated, and tied to one operator's local machine, Cloudflare account, and Upstash databases.\n\n## What We Are Building\n\nAt a high level, the system has five layers:\n\n1. Ingestion\n   It pulls raw material from AI conversation exports, coding-agent sessions, GitHub, and Gmail.\n2. Distillation\n   It turns raw material into structured memory entries with provenance.\n3. Storage\n   It stores those entries in Redis and their embeddings in Upstash Vector.\n4. Retrieval\n   It serves the memory through a Cloudflare-hosted MCP server with tier-aware search and thin-index summaries.\n5. Maintenance\n   It will promote, demote, consolidate, and archive memories over time through reconsolidation and Dream jobs.\n\nThe end state is a memory system that behaves less like a document archive and more like a selective autobiographical memory.\n\n## The Full System Vision\n\nThe target system has these behaviors:\n\n- durable identity and long-lived project context should be available by default\n- recurring but lower-priority context should appear when it is relevant\n- one-off facts should stay retrievable without constantly taking up context window space\n- repeated retrieval should strengthen memories\n- stale or low-value memories should be archived, not deleted blindly\n- nightly or scheduled \"Dream\" runs should reconcile the current self-model from accumulated experience\n\nThat full design is larger than what is currently live. Some parts are already running in production; some are the next phases.\n\n## Why Forgetting Matters\n\nThere is a specific kind of irritation that comes from watching an intelligent system confidently misread you.\n\nI was deep in a conversation about country equity rotation signals, the kind of technical discussion that represents the actual center of my professional life, when Claude helpfully suggested that I might want to think about the question through the lens of Ayurvedic medicine because I had apparently expressed interest in Panchakarma once, in a single conversation, three months earlier.\n\nI had asked one question. I did not need that interest preserved in perpetuity.\n\nThis is not Claude's fault. It is the fault of how AI memory systems are usually designed.\n\nMost deployed AI memory systems run on what I think of as single-gate ingestion:\n\n- did something appear in a conversation?\n- yes\n- store it forever\n- full weight\n- no decay\n- no frequency threshold\n\nThere is no mechanism to distinguish between something discussed across a hundred professional conversations spanning decades and something mentioned once on a slow Tuesday. Everything that clears the gate is treated as equally durable, equally salient, and equally worthy of surfacing later.\n\nThe brain does not work this way.\n\nModern memory research makes the point clearly: optimal memory is not maximal memory. During slow-wave sleep, the hippocampus replays recent experience selectively, not uniformly. Weak signals do not consolidate like strong ones. The Synaptic Homeostasis Hypothesis proposes that sleep globally downscales synaptic strength while preserving stronger connections proportionally better, improving signal-to-noise ratio. The goal of memory is not to retain everything. The goal is to improve future decisions.\n\nThat is the design principle behind this repository.\n\nThe system is being rebuilt around a simple premise:\n\n- forgetting is not failure\n- forgetting is how a memory system becomes useful\n\n### What The Old System Got Wrong\n\nBefore the current upgrade, the memory layer behaved too much like a write-only archive.\n\nEntries such as:\n\n- \"Ayurvedic medicine — Panchakarma therapy\"\n- \"NFL playoff psychology\"\n\ncould sit at effectively the same retrieval weight as:\n\n- \"30 years in quantitative investing\"\n- \"Senior Advisor at GMO\"\n\nThe system had no real evidence model. It had almost no way to distinguish durable identity from incidental curiosity. It also was not properly learning from use: access counters and last-accessed timestamps were not wired into retrieval behavior, so the store could accumulate entries without any feedback loop telling it which ones continued to matter.\n\nThe result was completeness without judgment.\n\n### Evidence Strength\n\nThe first structural change was to make the system aware of what it does not yet know: how strong the evidence is behind any given memory.\n\nEvery entry now carries a `context_type`, a classification of why the information is being stored. The taxonomy runs from stable identity and active project context down to task-query and passing-reference material.\n\nEvery entry also carries a `mention_count`, which tracks how many independent source conversations or source clusters surfaced that topic. This is the frequency signal the system uses to distinguish a durable pattern from a one-off appearance.\n\nFrom there, the system derives an `injection_tier`:\n\n- Tier 1\n  Stable identity, stated preferences, and active projects.\n- Tier 2\n  Recurring patterns and medium-priority topic-adjacent context.\n- Tier 3\n  Passing references and direct-query-only material.\n\nThe important property is that Tier 3 material does not need to vanish to stop being intrusive. It can remain fully retrievable while no longer surfacing uninvited.\n\n### The Salience Function\n\nEvidence strength is not binary. It is continuous, and it changes over time.\n\nThe salience model in this repository combines:\n\n- recency decay, with different half-lives by context type\n- mention frequency, with diminishing returns\n- type multipliers, so identity and project context are not treated like passing references\n- a small retrieval bonus, so repeatedly used memories can strengthen over time\n\nThis lets the system represent a practical difference between:\n\n- a core professional identity fact that should remain available indefinitely\n- a one-off query that should become functionally invisible unless reinforced\n\nThis is the software analog of strategic forgetting: preserve what matters, but stop letting weak traces compete for attention forever.\n\n### The Dream Job\n\nThe most ambitious part of the design is the Dream job.\n\nDream is live as a governed maintenance path. The scheduler, audit trail,\ncandidate discovery loop, proposal generation, deterministic grading, bounded\napply, rollback mechanics, deterministic duplicate merge, contradiction\nhandling, judge queue, and write-capable operator tools are implemented. The\nnightly Worker runs governed live apply directly from the schedule; it is not\ncontrolled by proposal-only/off-mode environment switches.\n\nThe intended structure is:\n\n1. Survey\n   Load active entries, compute salience, and bucket them into stable, active, weak, and decay candidates.\n2. Replay\n   Scan recent activity for repeated topics, contradictions, promotion candidates, and duplicates.\n3. Propose\n   Create a bounded mutation plan with evidence, expected revisions, and rollback metadata.\n4. Grade\n   Run deterministic hard gates before any live mutation is allowed.\n5. Apply And Verify\n   Apply only explicitly approved operations, then verify index/search/vector consistency.\n6. Roll Back\n   Restore from apply artifacts when current revisions still match the expected post-apply state.\n\nThe point of Dream is not to erase the past. The point is to keep the active memory layer aligned with what remains useful.\n\n### Reconsolidation On Retrieval\n\nThe shorter-horizon companion to Dream is reconsolidation on retrieval.\n\nThis is now live. The behavior is straightforward:\n\n- every meaningful retrieval becomes a write event\n- access count increments\n- last accessed updates\n- the system can reconsider whether the memory has earned a stronger context type or tier\n\nIn other words, use matters. A memory that continues to be retrieved across different conversations should not be treated the same way as one that was never touched again after ingestion.\n\n### The Larger Point\n\nThe problem here is not specific to Claude. It is a design philosophy problem that shows up across AI memory systems.\n\nMost systems are optimized for recall coverage:\n\n- do not miss anything\n- do not forget anything\n- preserve everything forever\n\nBut a memory system that never forgets is not automatically intelligent. It may simply be a system that traded judgment for completeness.\n\nThe brain solved this a long time ago:\n\n- fast encoding\n- slow integration\n- selective replay\n- active weakening of noise\n- durable retention for what survives repeated relevance\n\nThat is the model this repository is trying to approximate.\n\nThe goal is not a perfect record.\n\nThe goal is a memory system that gets better at representing what matters.\n\n## System Model\n\nCurrent high-level schema:\n\n![Personal Knowledge System architecture](docs/assets/pks-architecture-schema.png)\n\n```text\nRaw Sources\n  Claude exports\n  ChatGPT exports\n  GitHub repos\n    README / commits / code comments\n    repo-attached Claude Code / Codex CLI / Cursor context\n  Gmail\n\n        |\n        v\n\nIngestion + Distillation\n  Python pipelines extract structured entries\n  Models assign provenance, summaries, and embeddings\n  Migration/backfill scripts normalize old data\n\n        |\n        v\n\nMemory Store\n  Upstash Redis\n    knowledge:{id}\n    project:{id}\n    index:current\n    migration flags\n    future Dream/reconsolidation state\n\n  Upstash Vector\n    one embedding per active entry\n    metadata for retrieval filters and scoring\n\n        |\n        v\n\nRetrieval Layer\n  Cloudflare Worker MCP server\n  OAuth-enabled public interface\n  thin index\n  semantic search\n  context retrieval\n  health/status endpoint\n\n        |\n        v\n\nFuture Maintenance Layer\n  reconsolidation on repeated access\n  Dream coordinator and archive pipeline\n  operator tools for restoration and overrides\n```\n\n## Memory Model\n\nThe system stores two primary entry types:\n\n- `KnowledgeEntry` (`ke_*`)\n  A durable belief, skill, preference, technique, or topic model.\n- `ProjectEntry` (`pe_*`)\n  An ongoing effort with goals, status, phase, blockers, and decisions.\n\nEach entry has a `schema_version` and migration-safe metadata. The important fields in the current design are:\n\n- `context_type`\n  What kind of memory this is: identity, project, pattern, task-query, and so on.\n- `injection_tier`\n  How aggressively this memory should be surfaced.\n- `salience_score`\n  A score derived from confidence, recency decay, mention frequency, context type, and recent retrieval.\n- `classification_status`\n  Whether the entry has been backfilled/classified yet.\n- `archived`\n  Whether the entry should be excluded from normal retrieval.\n\n### Retrieval Tiers\n\nThe system is moving toward a three-tier memory model:\n\n- Tier 1\n  Durable identity, long-running projects, and context that should often be available.\n- Tier 2\n  Recurring, topic-adjacent, or medium-priority context.\n- Tier 3\n  One-off or direct-query-only context that should stay searchable but not dominate context injection.\n\nThis tiering is the main mechanism for preventing the memory system from becoming a pile of equally weighted notes.\n\n## Retrieval Model\n\nThe production MCP server exposes:\n\n- `get_index`\n  Returns the thin-index subset plus true totals and tier counts.\n- `get_context`\n  Returns the current view of the best matching active topic or project.\n- `get_deep`\n  Returns the full stored entry with provenance.\n- `search`\n  Performs tier-aware semantic retrieval.\n- `github`\n  Queries linked GitHub repositories live.\n\nSearch no longer uses a simple \"70% relevance + 30% recency\" rule. The current design scores and ranks results using:\n\n- semantic similarity\n- recency\n- salience\n- source weights\n- retrieval tier\n\nArchived entries remain in storage but are excluded from normal retrieval by default.\n\n## Thin Index\n\nThe thin index is the compressed map of the memory system.\n\nIt is intentionally not a full dump of every entry. It stores:\n\n- a tier-stratified subset of topics and projects (up to 100 total: 15 T1 slots, 25 T2 slots, 60 T3 slots)\n- true total topic/project counts\n- tier counts\n- archive counts\n- recent evolution summaries (carried forward from the prior index)\n\nThis lets a client get a fast overview of the memory landscape without paying the cost of loading the entire store.\n\n## Dream And Reconsolidation\n\nThese are the main pieces still being built.\n\n### Reconsolidation\n\nReconsolidation is the short-horizon maintenance loop. The idea is:\n\n- retrieval increments access counters\n- frequently re-accessed memories get promoted or refreshed\n- repeated retrieval can strengthen salience\n- the system records consolidation notes and errors\n\nThis is Phase 4 work and is live.\n\n### Dream\n\nDream is the long-horizon maintenance loop. The idea is:\n\n- run on a schedule\n- revisit the memory graph in batches\n- keep durable context\n- archive low-value memories with reversible pointers\n- rebuild the current self-model without re-injecting everything forever\n\nDream is Phase 5 work. The live scheduler, audit output, proposal artifacts, deterministic grading, reversible apply/rollback path, judge queue, and write-capable operator tools are implemented. The deployed scheduler now runs governed live maintenance with policy-driven archive and promotion caps loaded from `shared/memory_policy.json`. Scheduled Dream is expected to record `dry_run=false`, `trigger=scheduled`, and `auto_apply_mode=governed`.\n\n### Current Scheduling Model\n\n> **In migration (2026-06-16):** the multi-scheduler model below is being replaced\n> by a single M4-owned production controller — see\n> `docs/pks-nightly-orchestrator-redesign-2026-06-16.md`. The new orchestrator\n> (`orchestrator/` package, entrypoint `scripts/nightly_orchestrator.py`) is being\n> built in phases and is **shadow-only / non-mutating** until cutover. The\n> schedulers listed here remain the production source of truth until Phase 5.\n\nThe system currently uses three schedulers for three different jobs:\n\n- GitHub Actions runs Twitter ingestion remotely at `05:40 UTC`\n- GitHub Actions runs GitHub ingestion, including repo-attached agent context, at `06:10 UTC`\n- Cloudflare Workers runs the remote Dream job at `07:10 UTC`\n\nRepo-attached AI context is exported locally at commit time into\n`.pks/agent-context/` and then picked up by the nightly GitHub ingestion job.\nThat keeps the automated nightly path remote while still associating the chat\nhistory with the repository itself. The workflow lives at\n`.github/workflows/github-ingestion.yml`.\n\nTwitter ingestion is scheduled remotely because the source data comes from the\nX API rather than local files. The workflow lives at\n`.github/workflows/twitter-ingestion.yml` and uses a Redis-backed checkpoint so\nscheduled runners do not lose incremental state when the GitHub Actions runner\nis destroyed.\n\nLocal machine ingestion still handles local-only AI session sources. The\nlaunchd wrapper is `scripts/run_nightly_ingestion.sh`; it sources the repo\nroot `.env`, expands launchd's minimal `PATH`, verifies the ingestion venv,\nand fails fast if the Claude CLI cannot be found.\n\n#### Nightly resilience (2026-06-10 re-architecture)\n\n- **Upstash storage preflight**: before any stage runs, the wrapper polls the\n  Redis + Vector REST endpoints every 2 minutes for up to 30 minutes. A\n  transient cloud outage at 23:00 becomes a short delay, not a dead night.\n- **Per-stage retry**: a hard-failed stage is retried once after 5 minutes\n  (`NIGHTLY_STAGE_RETRY_DELAY` to tune). Exit code 2 (partial) is not retried.\n- **PARTIAL/WARN semantics**: `ingestion/github/run.py` exits **2** when some\n  repos failed extraction but all successful entries were saved (failed repos\n  auto-retry next night). The night is recorded as OK-with-warnings\n  (`warn_stages` in `nightly_ingestion_success.json`), not FAILED. Only hard\n  failures (storage down, save failed) fail the night.\n- **02:00 second chance**: `scripts/run_second_chance.sh`\n  (launchd `com.arjun.knowledge-ingestion-2am`) re-runs the whole idempotent\n  ingestion if the 23:00 run failed or never completed.\n- **07:00 NightWatch digest**: `/Users/arjundivecha/Dropbox/AAA Backup/A\n  Working/NightWatch/night_supervisor.py` reads the success marker and sends a\n  consolidated morning iMessage covering this and every other overnight system.\n\n#### Claude Billing Route (SDK primary, API fallback, never skip)\n\nBoth the local wrapper and all three ingestion workflows\n(`twitter-ingestion.yml`, `github-ingestion.yml`,\n`agent-session-ingestion.yml`) select a Claude billing route before fetching\nany source data:\n\n1. A non-interactive preflight,\n   `scripts/check_claude_sdk_auth_noninteractive.py`, probes whether Claude\n   Agent SDK subscription auth works in this process.\n2. If it does, ingestion uses the SDK subscription route\n   (`PKS_ALLOW_ANTHROPIC_API_FALLBACK=0`).\n3. If it does not, ingestion routes to the project Anthropic API key as a\n   bounded fallback (`PKS_ALLOW_ANTHROPIC_API_FALLBACK=1`, with per-call and\n   per-run budget/call caps). Ingestion is **never** silently skipped, and the\n   wrapper fails loudly if fallback is needed but no `ANTHROPIC_API_KEY` exists.\n\nThe preflight is deliberately **no-browser**. A self-hosted runner or launchd\njob has no interactive session, so a naive SDK auth probe could start an OAuth\nlogin *browser* flow (a listener on `localhost:18043`, `oauth/callback` URLs)\nand, when several jobs run at once, multiply into a session storm. The wrapper\nhardens the child environment (`CI=1`, `BROWSER=/usr/bin/false`,\n`GIT_TERMINAL_PROMPT=0`, and scrubs `ANTHROPIC_API_KEY` so it tests\nsubscription auth specifically), runs the probe in its own process group under\na hard timeout, and SIGKILLs that group if it stalls. The workflows and the\nlaunchd wrapper also export `BROWSER=/usr/bin/false` for the whole run as a\nsecond guard. The Dream judge still uses the Claude subscription CLI path and\ndoes not enable API fallback unless `DREAM_ALLOW_ANTHROPIC_API_FALLBACK=1` is\nset explicitly.\n\nCloudflare Worker deploys should use:\n\n```bash\nbash scripts/deploy_cloudflare_worker.sh\n```\n\nThe script sources `.env`, requires `CLOUDFLARE_API_TOKEN`, and deploys the\nproduction Worker with `npx wrangler deploy --env=\"\"`.\n\n## What Is Live Today\n\nAs of June 3, 2026, the live system has:\n\n- `4,620` total topics in the thin-index accounting\n- `34` active projects\n- schema version `2`\n- completed Phase 1, Phase 2, Phase 3, and Phase 4 of the current memory upgrade\n- `0` pending classifications in `classification:pending`\n- thin-index counts of `887` Tier 1, `1,125` Tier 2, `2,642` Tier 3\n- `4,968` archived entries at the time of the latest health check\n- latest deployed Dream code is governed-live scheduled maintenance\n- the Dream judge queue is empty in production\n- strict overnight checks require the latest post-cron scheduled Dream run to be governed/live, not proposal-only\n\nOperationally, the following are live:\n\n- Python ingestion/distillation pipelines\n- shared salience policy between Python and TypeScript\n- vector metadata normalization\n- rebuilt thin index with tier/salience metadata\n- OAuth-enabled Cloudflare MCP server\n- background reconsolidation on retrieval\n- write-capable MCP tools for `restore_archived` and `set_context_type`\n- remote scheduled Twitter/X ingestion with Redis-backed incremental state\n- remote scheduled GitHub ingestion with repo-attached Claude Code, Codex CLI, and Cursor context\n- local nightly ingestion for local-only AI session logs, with Redis-backed checkpointing and local checkpoint fallback\n- nightly governed-live Dream scheduler and audit records\n- deterministic Dream proposal grading\n- bounded Dream apply and conflict-aware rollback mechanics\n- deterministic duplicate merge and contradiction handling in Dream replay\n- Dream judge queue with local Claude CLI verdicting\n- `/health` and `/status` rollout endpoints\n\nNot live yet:\n\n- external-runner fallback for heavier Dream work\n\n## How The Repo Is Organized\n\n```text\nknowledge-system/\n  ingestion/\n    github/, gmail/, agent_sessions/\n    Python ingestion pipelines for ongoing raw-source intake\n\n  distillation/\n    Original export-processing pipeline for Claude/ChatGPT data\n    Also contains storage clients, models, and thin-index generation\n\n  scripts/\n    Migration, verification, ingestion-billing, and health-monitor scripts\n    backfill_context_type.py\n    backfill_counts.py\n    verify_memory_consistency.py\n    check_claude_sdk_auth_noninteractive.py   no-browser SDK auth preflight\n    nightly_health_monitor.py                 preflight / snapshot / verify\n    oauth_storm_watcher.sh                    OAuth storm tripwire (detect-only)\n\n  shared/\n    Cross-language policy files\n    memory_policy.json\n    salience_fixtures.json\n\n  cloudflare-mcp/mcp-server/\n    Production MCP server\n    Cloudflare Worker, OAuth wrapper, retrieval tools\n\n  mcp-server/\n    Legacy server implementation\n    Not the production target\n\n  docs/\n    PRDs, audit notes, and upgrade checklists\n\n  skill/\n    Claude skill instructions for using the memory system\n```\n\n## Operational Surfaces\n\n### Worker Endpoints\n\nThe public Worker exposes:\n\n- `/sse`\n- `/mcp`\n- `/authorize`\n- `/token`\n- `/register`\n- `/.well-known/oauth-authorization-server`\n- `/health`\n- `/status`\n\n### Health Endpoint\n\n`/health` and `/status` are the main operator-facing rollout checks. They report:\n\n- schema version\n- migration completion state\n- pending classification count\n- last Dream run timestamp\n- thin-index totals\n- tier counts\n- archived count\n\n### Dream Run Checks\n\nThe strict overnight check is:\n\n```bash\nset -a; source .env; set +a\ningestion/.venv/bin/python scripts/check_overnight_dream_run.py\n```\n\nIt validates the latest scheduled Dream run against the policy source of truth\nin `shared/memory_policy.json`. A passing post-cron run must be scheduled,\ngoverned, live (`dry_run=false`), and within the expected time window.\nIt reads the Dream run ledger rather than only `dream:last_run`, so a cautious\nfully-held governed run is still visible to the check. Each run writes both a\nJSON diagnostic and a Markdown sleep report:\n\n```bash\nscripts/reports/check_overnight_dream_run_*.json\nscripts/reports/dream-sleep-YYYY-MM-DD.md\n```\n\nGitHub Actions also runs `.github/workflows/nightly-sleep-report.yml` nightly at\n08:45 UTC and uploads those sleep-report artifacts after the Worker Dream cron.\n\nThe live judge queue can be inspected through the operator endpoint:\n\n```bash\nset -a; source .env; set +a\ncurl -sS -H \"Authorization: Bearer $DREAM_OPERATOR_TOKEN\" \\\n  https://mcp.dancing-ganesh.com/ops/dream/judge_queue\n```\n\n### Migration Scripts\n\nThe upgrade work introduced three important operator scripts:\n\n- `scripts/backfill_context_type.py`\n  LLM classification pass for old entries.\n- `scripts/backfill_counts.py`\n  Deterministic metadata/vector normalization and thin-index rebuild.\n- `scripts/verify_memory_consistency.py`\n  Redis vs Vector vs thin-index verification.\n\nThese scripts are how the repo moved from legacy mixed-schema data to the current retrieval model.\n\n### Nightly Ingestion Health Monitoring\n\n`scripts/nightly_health_monitor.py` is the end-to-end health check for the\nnightly ingestion run. It is read-only against production storage and has three\nmodes:\n\n```bash\n# Readiness checks (env keys, source dirs, claude CLI, agent-sdk import,\n# Redis/Vector connectivity, and the no-browser SDK preflight). No browser opens.\ningestion/.venv/bin/python scripts/nightly_health_monitor.py preflight\n\n# Capture observable state (storage counts, per-source dedup counts, checkpoints)\n# once before and once after a run.\ningestion/.venv/bin/python scripts/nightly_health_monitor.py snapshot --label before --out /tmp/before.json\ningestion/.venv/bin/python scripts/nightly_health_monitor.py snapshot --label after  --out /tmp/after.json\n\n# Per-pipeline PASS/WARN/FAIL report (also scans the run log for errors and any\n# OAuth/browser-storm references, scoped to the most recent run in the daily log).\ningestion/.venv/bin/python scripts/nightly_health_monitor.py verify \\\n  --before /tmp/before.json --after /tmp/after.json \\\n  --log ingestion/logs/nightly/$(date +%F).log --out /tmp/nightly_report\n```\n\nA clean incremental run reports `WARN` when a pipeline simply found nothing new;\n`FAIL` means the run did not complete, the log contained errors, or any\nbrowser/OAuth-storm reference appeared.\n\n`scripts/oauth_storm_watcher.sh` is a **detect-and-alert-only** tripwire meant\nto run in the background alongside a nightly run. It polls for a new (non-baseline)\nlistener on port `18043` or an `oauth/callback` process and logs an `ALERT` if it\nsees one. It deliberately does **not** kill anything: port `18043` is also used\nlegitimately by the `mcp-remote` knowledge-Worker client and by the no-browser\npreflight's own sandboxed probe, so killing risks taking down legitimate clients\n(or the run itself). The storm is prevented at the source by the no-browser\npreflight; this watcher is just a secondary tripwire. Stop it by creating\n`<log>.stop` or sending `SIGTERM`. The verification logic and the no-browser\npreflight are covered by `tests/python/test_nightly_health_monitor.py` and\n`tests/python/test_check_claude_sdk_auth_noninteractive.py`.\n\n### Nightly Fault Isolation\n\n`scripts/run_nightly_ingestion.sh` runs each pipeline (Twitter, GitHub, Agent\nsessions, Dream judge) as an isolated **stage**: a stage failure is logged and\nrecorded but never aborts the remaining stages. (Previously a single repo's\nmalformed-JSON README aborted GitHub, and `set -e` then skipped agent-sessions\nand Dream.) The success marker now always records per-stage exit codes plus an\n`ok` flag and a `failed_stages` list, and the wrapper exits non-zero if any\nhard stage (Twitter/GitHub/Agent sessions) fails — Dream judge is a tolerated\nsoft stage. Relatedly, GitHub extraction isolates per-repo failures: a repo\nwhose extraction fails is left unmarked (retried next run) while the repos that\nsucceeded are still saved, and LLM JSON output is parsed tolerantly via\n`json-repair` so a single malformed model response is repaired rather than\nfatal.\n\n## Validation Strategy\n\nThis repo now has an explicit testing plan. The goal is to validate the full memory loop, not just confirm that individual scripts exit successfully.\n\nThe three operating layers are:\n\n- fixture tests for deterministic logic and policy behavior\n- staging end-to-end tests against isolated Redis, Vector, and Worker infrastructure\n- production canaries for bounded live verification only\n\nThe testing system is documented in [docs/testing-matrix.md](/Users/arjundivecha/Dropbox/AAA%20Backup/A%20Working/Memory/knowledge-system/docs/testing-matrix.md).\n\nThe root [Makefile](/Users/arjundivecha/Dropbox/AAA%20Backup/A%20Working/Memory/knowledge-system/Makefile) is the command surface for this work. The current starter commands are:\n\n- `make worker-typecheck`\n- `make worker-test`\n- `make test-python-checker`\n- `make verify-memory-full`\n- `make check-overnight-dream`\n- `make seed-staging-dry-run`\n- `make staging-smoke-dry-run`\n- `make staging-smoke`\n- `make worker-secrets-staging`\n- `make deploy-staging`\n- `make dream-live-canary`\n\n### Retrieval-quality eval baseline (added 2026-07-06)\n\n`scripts/run_eval.py` runs the probe suite in [tests/probes/](tests/probes/README.md)\n(8 axes: recall, project, explicit-save, exact-lexical, stale-fact, supersession,\nnegative, paraphrase) read-only against production or staging and writes\n`scripts/reports/eval_baseline_<UTC>.json`. `--compare OLD NEW` diffs two reports\n(the shadow A/B safety rail for retrieval changes). Design and metric definitions:\n`../PRD-eval-baseline-v1.md`. Rule: no ranking/forgetting/admission change ships\nwithout a before/after eval diff.\n\nFirst production baseline (2026-07-06, 51 probes): carry_forward_recall 0.90,\nproject_recall 0.86, exact_lexical 0.67, stale_leak_rate 0.25, supersession 0.50,\nnegative_precision 1.00, paraphrase_consistency 0.67, ~254 tokens/query median.\nThe lexical and stale/supersession gaps are the expected targets of upgrade-plan\nitems 1 and 5. The LLM-judge for future answer-mode scoring is DeepSeek V4 Flash\nvia the direct API (validated 26/26 vs Sonnet 4.6 and V4 Pro; escalation to Sonnet\non uncertainty).\n\nThe long-term rule is simple: production is not the default test bed.\n\nThe staging smoke path is now governance-shaped. It covers:\n\n- fixture seeding into isolated staging Redis and Vector\n- staging Worker `/health`\n- unauthorized operator rejection\n- Dream proposal generation on staging\n- deterministic Dream proposal grading\n- bounded proposal apply on staging\n- post-apply verification\n- conflict-aware rollback on staging\n- post-rollback verification\n- validation-ledger recording for `staging_e2e`\n- `/openai/mcp` read-only compatibility\n- OAuth discovery, client registration, auth-code exchange, and bearer-token issuance\n- MCP `initialize`, `tools/list`, `get_index`, `search`, `get_context`, and `get_dream_summary`\n- final Redis vs Vector vs thin-index consistency verification\n\nThere is now also a local Worker-runtime test layer under [cloudflare-mcp/mcp-server/test](/Users/arjundivecha/Dropbox/AAA%20Backup/A%20Working/Memory/knowledge-system/cloudflare-mcp/mcp-server/test). It runs inside Cloudflare's `workerd` runtime and covers:\n\n- `/health`\n- unauthorized operator rejection\n- OAuth discovery, client registration, auth-code exchange, and token issuance\n- MCP `initialize`, `tools/list`, `get_index`, and `get_dream_summary`\n- write-tool rejection without `mcp:write`\n- scheduled governed-live Dream trigger wiring\n\nThat local test layer is automated in GitHub Actions at [.github/workflows/worker-runtime-tests.yml](/Users/arjundivecha/Dropbox/AAA%20Backup/A%20Working/Memory/knowledge-system/.github/workflows/worker-runtime-tests.yml).\n\n## Current Upgrade Status\n\nThe repository is in the middle of a larger PKS memory upgrade.\n\nCompleted:\n\n- Phase 0 audit and gap analysis\n- Phase 1 schema and migration hooks\n- Phase 2 live backfill and normalization\n- Phase 3 tier-aware retrieval and rollout status endpoint\n- Phase 4 reconsolidation on retrieval\n- Phase 5 Dream control plane, archive/restore path, replay heuristics, and staging validation\n- Phase 6 operator tool surface and write-scope enforcement\n\nNext:\n\n- richer Dream replay logic beyond the current deterministic duplicate/contradiction heuristics\n- Phase 7 evidence log and compiled-view separation, so contradictions can be resolved at the source while superseded claims remain preserved as evidence\n- ingestion hardening and source-fusion improvements\n- broader fixture and ranking coverage in the automated test stack\n\nThe upgrade checklist lives in `docs/pks-memory-upgrade-checklist.md`.\n\n## Important Reading Order\n\nIf you are trying to understand the system, read in this order:\n\n1. this README\n2. `docs/pks-memory-upgrade-checklist.md`\n3. `docs/pks-memory-upgrade-phase0-audit-2026-03-26.md`\n4. `cloudflare-mcp/mcp-server/src/index.ts`\n5. `distillation/models/entries.py`\n6. `distillation/pipeline/index.py`\n7. `shared/memory_policy.json`\n\nThat path gives the clearest picture of the actual architecture and the upgrade trajectory.\n\n## Design Principles\n\nThe system is trying to enforce a few simple rules:\n\n- memory should be selective, not exhaustive\n- provenance matters\n- retrieval quality matters more than raw storage volume\n- the system should prefer reversible archival over destructive cleanup\n- scoring rules should be shared across languages and runtimes\n- health and migration state should be observable, not implicit\n\n## Version History\n\n- **1.2.2** (June 2026)\n  Nightly reliability: fault-isolated stages (one pipeline failure no longer\n  aborts the night), tolerant `json-repair` parsing of LLM extraction output,\n  per-repo isolation in GitHub ingestion (save good repos, retry failed ones),\n  success marker with per-stage status / `ok` / `failed_stages`, sturdier\n  agent-session Redis mirror retries, and a safer detect-only OAuth storm\n  watcher.\n- **1.2.1** (June 2026)\n  No-browser Claude SDK auth preflight (`check_claude_sdk_auth_noninteractive.py`)\n  to stop the ingestion OAuth/browser session storm; SDK-primary / API-fallback /\n  never-skip billing route across all three ingestion workflows and the launchd\n  wrapper; end-to-end nightly health monitor and OAuth storm watcher with tests.\n- **1.2.0** (March 2026)\n  Schema v2 migration, context-type backfill, tier-aware retrieval, shared salience policy, `/health` endpoint, OAuth-enabled Worker deployment.\n- **1.1.0** (March 2026)\n  Agent session ingestion, GitHub repo linking, remote scheduling transition groundwork, model upgrade to Claude Sonnet 4.6.\n- **1.0.1** (March 2026)\n  GitHub and Gmail ingestion pipelines, recency weighting, source-based scoring, thin-index compaction.\n- **1.0.0** (December 2024)\n  Initial implementation with distillation pipeline, Cloudflare MCP server, and Claude integration.\n\n## Current Upgrade Program (2026-07)\n\n`docs/pks-foundational-upgrade-spec-2026-07-07.md` is the active foundational\nupgrade spec (Fable 5 commission): Phase 0 diagnosis of the live store,\ninjection/ranking redesign, consolidation + lossless-merge gating, contradiction\nresolution, evaluation harness, and a staged reversible rollout. Its executable\nhalf lives in `contracts/*.spec.md` — seven Divecha-gated implementation\ncontracts (author-validated) meant to be handed to cheaper coding models in\nBuild Mode. Start with `contracts/retrieval-regression-gate.spec.md` (stage 0),\nthen `usage-signal-loop`, per the rollout table in the spec §6.\n\n## License\n\nPrivate repository. Not for redistribution.\n"}}}}