{"repo": "axoviq-ai/synthadoc", "files": {"docs/design.md": {"eb23d9720969400c5d3ad25bda049b5aa3e65817": {"t": 1785746718.2393916, "n": 229186, "h": "783794f6497010c7", "c": "# Synthadoc — Design Document\n\n**Version:** 1.1.3 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n29. [Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer)\n30. [Per-Source Truncation Flag](#30-per-source-truncation-flag)\n31. [Proportional Context Budget](#31-proportional-context-budget)\n32. [Knowledge Graph](#32-knowledge-graph)\n33. [Multi-Platform Agent Skill Files](#33-multi-platform-agent-skill-files)\n34. [Synthadoc Scaffold Marker](#34-synthadoc-scaffold-marker)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for Codex CLI, OpenCode, and generic OpenAI agents\n CLAUDE.md ← same content as AGENTS.md; loaded automatically by Claude Code\n GEMINI.md ← same content as AGENTS.md; loaded automatically by Gemini CLI\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the full source text (bounded by `max_source_chars`) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `flag` / `skip`). Pages with `status='active'` are protected — sources that conflict with them trigger `flag` rather than `update` (RULE 1b).\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Orchestrator calls `_guard_ingest_cost()` on the returned cost: soft warn → `logger.warning` only; hard gate → `cost_gate_exceeded` audit event + `fail_permanent` (job → DEAD). On success, job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 0b — Key Data Extraction | None (regex) | Deterministic pre-processor: extracts numbers, percentages, currency amounts, ratios, formulas, underscore identifiers, and date ranges from source text via regex; appends them as a `[Key Data — extracted by pre-processor]` section before any LLM sees the text. Zero token cost; cannot hallucinate. Anchors the synthesis LLM to all quantitative candidates. |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads the full source text (bounded by `max_source_chars`) + BM25 candidates + `purpose.md` scope + status of each candidate page. Outputs per-page action: `create`, `update`, `flag`, `skip`. **RULE 1b:** a page with `status='active'` is human-reviewed and authoritative — when the source provides a conflicting value, date, formula, or conclusion, the action must be `flag`, not `update`. The only exception is a source that adds a topic the page does not yet mention at all. Decision cache version: `v2`. |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. If no citation markers are present in the returned body, a `citation_pass4_no_markers` WARNING audit event is written — the page is still saved but the condition is flagged for re-ingest with a more capable model. Zero-citation results are not cached so a subsequent re-ingest can succeed. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic. **Suppressed when TF fallback was used** (a small corpus with broad coverage may legitimately return fewer than 3 candidates — that is expected, not a gap).\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n| Citation presence (Check 5b) | Page body has ≥ 50 words and zero `^[filename:L-L]` citation markers → WARNING. Diagnostic only — does not block promotion to `active`. Indicates the annotation pass failed to produce markers, usually a model-compatibility issue (use Gemini 2.5 Flash or higher for reliable citation annotation). |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n### ActionAgent\n\nDispatches action-intent queries from the chat UI (e.g. \"activate a draft page\", \"show lint report\", \"archive a stale page\", \"run scaffold\"). Parses the user's free-text intent into a structured action via an LLM extraction prompt, then executes the action against the wiki and returns a human-readable result. When the intent is ambiguous (e.g. \"activate a page\" without specifying which), sets `needs_clarification=True` and returns a prompt and candidate list for the web UI to render as chip buttons.\n\n### ScaffoldAgent\n\nGenerates and updates the wiki's structural pages. Reads all current wiki pages and asks the LLM to organise them into 5–8 domain-appropriate categories, then writes `wiki/index.md` (category index with wikilinks), `wiki/purpose.md` (scope statement), and `AGENTS.md` (ingest and query guidelines). Also stamps a `categories:` field on each page's YAML frontmatter. When `ROUTING.md` already exists, regenerates it to stay in sync with the updated index structure.\n\n### RewriteAgent\n\nRewrites follow-up questions in multi-turn conversations into self-contained, standalone form before BM25 retrieval. Converts context-dependent phrases (\"What came after that?\", \"Tell me more about his early life\") into explicit questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the correct pages without relying on conversation context. Only invoked when conversation history is non-empty. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### SummarizeAgent\n\nCompresses the oldest conversation turns into a single `[Session summary: …]` assistant turn when a session exceeds `conversation_history_turns`. Prevents unbounded context growth across long sessions. Emits a `notice` SSE event the first time compression occurs so the user can see that earlier context was condensed. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### ContextAgent\n\nBuilds token-budgeted context packs for MCP tool consumers. See [Section 20 — Context Packs](#20-context-packs) for full detail.\n\n### SearchDecomposeAgent\n\nDecomposes a web search intent into 1–4 terse keyword search strings optimised for authoritative source retrieval. Uses a separate prompt from `QueryAgent`'s question decomposition — search decomposition asks \"what distinct search strings would find the best sources?\" (terse keyword phrases) rather than \"what distinct questions does this ask?\" (natural-language sub-questions). See [Web search fan-out](#web-search-fan-out) in the IngestAgent section.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n| `session` | `.jsonl` | `claude session`, `codex session`, `cursor session`, `ai session`, `session history` | Extracts human-readable turns from AI coding session transcripts. Supports Claude Code JSONL format and Codex/Cursor format. Filters tool calls, thinking blocks, and sub-agent scaffolding. Short turns (< 20 assistant words, < 3 user words) are skipped. No external dependencies. |\n\n### Session Skill — AI Session History Ingestion\n\nThe `session` skill turns AI coding session history files (`.jsonl`) into searchable wiki pages capturing the problem-solving knowledge exchanged during the session.\n\n**Supported formats**\n\n| Format | File origin | Detection |\n|--------|-------------|-----------|\n| Claude Code | `~/.claude/projects//.jsonl` | `obj.type` ∈ `{\"user\", \"assistant\"}` with nested `obj.message` |\n| Codex / Cursor | Export from OpenAI Codex or Cursor IDE | `obj.role` at top level, no `message` wrapper |\n\nFormat is auto-detected from the first 30 parseable lines. Files that match neither format are parsed as Codex (fallback).\n\n**Filtering rules**\n\n| Content | Kept? | Reason |\n|---------|-------|--------|\n| User text ≥ 3 words | ✓ | Substantive question or instruction |\n| Assistant text ≥ 20 words | ✓ | Substantive answer |\n| User text < 3 words (`\"ok\"`, `\"yes\"`) | ✗ | Too terse to be useful |\n| Assistant text < 20 words | ✗ | Acknowledgement or one-liner |\n| `isSidechain: true` lines | ✗ | Sub-agent scaffolding (not the user's conversation) |\n| `tool_use` blocks | ✗ | Shell commands / file writes — not final output |\n| `tool_result` blocks | ✗ | Raw output — avoids leaking file contents or credentials |\n| `thinking` blocks | ✗ | Internal reasoning — not the final answer |\n| `permission-mode`, `system`, `last-prompt` lines | ✗ | Session metadata, not conversation |\n\nAfter extraction the text passes through Synthadoc's standard pre-LLM source sanitizer\n(zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars,\ninstruction-override phrases) — the same step applied to every PDF, DOCX, URL, and other source type.\nSee [§29 Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer).\n\n**Output format**\n\nEach kept turn is labelled and separated by `---`:\n\n```\n[USER]\nHow do I implement a sliding window algorithm in Python?\n\n---\n\n[ASSISTANT]\nA sliding window algorithm maintains a contiguous subarray by advancing two\npointers simultaneously…\n```\n\n**`suggested_slug`**\n\nThe skill sets `metadata[\"suggested_slug\"]` to `session-YYYY-MM-DD-` where the date comes from the file's `mtime` and the topic from the first substantive user turn (first 6 words, slugified). Example: `session-2026-07-15-how-do-i-implement-sliding`.\n\n**Limitations**\n\n- No chunking in v1.1 — very long sessions are truncated at `max_source_chars` (default 400 000 chars). Split large archives into individual `.jsonl` files.\n- Tool output is excluded by design — re-ingest the original source files if you need the file contents in the wiki.\n- Re-ingesting the same `.jsonl` file uses Synthadoc's standard source-hash dedup — no duplicate pages are created.\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n| `content_snapshot` | TEXT | Full page body at the moment of the event; NULL when the event carries no content change (e.g. ingest-agent transitions). See [§23 Page Content Snapshots](#page-content-snapshots) for the full capture policy. |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n**Compound identifier expansion:** When a token contains underscores, the tokenizer emits both the compound form and each component as separate tokens (`capex_growth` → `[\"capex_growth\", \"capex\", \"growth\"]`). This expansion is applied at both index time and query time so that human-friendly queries match programmatic identifiers and vice versa. Leading/trailing underscores are stripped before splitting; no empty strings are emitted.\n\n**TF fallback for small corpora:** When all BM25 scores in a result set are zero or negative (IDF collapse on a corpus of ≤ 2 documents, or a term present in every page), the search system falls back to normalised term frequency (TF) scoring: `TF(doc, query) = Σcount(term, doc) / len(doc_tokens)`. Results are returned in the same format as BM25 results with a `tf_fallback: True` flag on each `SearchResult`. This field is recorded in the audit trail and consumed by gap detection (see below).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp, cascade_links_removed_from: [str]}` _(cascade field added v1.0.2)_ |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n| `GET` | `/graph` _(v1.0.0)_ | — | `{status, node_count, edge_count, cluster_count, nodes: [...], edges: [...]}` or `{status: \"computing\"}` on first call |\n| `GET` | `/pages/{slug}/history` | `?index=N&include_content=true` | List content snapshots; ?index=N&include_content=true for single |\n| `POST` | `/pages/{slug}/rollback` | `{index: int, reason: str}` | Restore page body to snapshot N |\n| `POST` | `/pages/{slug}/snapshot` | `{content: str, reason?: str}` | Record a content snapshot only when content differs from the last stored snapshot (`recorded: true/false`). Called by the Obsidian plugin on vault modify events. |\n| `DELETE` | `/pages/{slug}/history` | — | Delete all lifecycle events (including snapshots) for a slug; intended for test teardown. |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n**External file paths** (outside the wiki root) are supported when the request comes from\n`127.0.0.1` or `::1` and the payload includes `\"allow_external_paths\": true`. The CLI sets\nthis flag automatically for all local file sources. Remote clients cannot set this flag\n(the server ignores it for non-localhost connections) to prevent arbitrary file reads\nin server-exposed deployments.\n\nThis is required for ingesting files that live outside the wiki directory by design —\nfor example, Claude Code session transcripts at `~/.claude/projects//.jsonl`.\n\n**Client behaviour summary:**\n\n| Client | Sends `allow_external_paths` | Can ingest outside wiki root? |\n|--------|------------------------------|-------------------------------|\n| CLI (`synthadoc ingest `) | `true` (automatic, local file paths only) | Yes — server is localhost |\n| Obsidian plugin | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Web UI | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Direct HTTP API (localhost) | Set manually in request body | Yes — if request from 127.0.0.1/::1 |\n| Direct HTTP API (remote) | Ignored by server | No — server silently treats as `false` |\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 1.1.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n**Reading View default:** `synthadoc plugin install` and `synthadoc plugin upgrade` write `\"defaultViewMode\": \"preview\"` to `.obsidian/app.json` after copying the plugin files. This causes Obsidian to open new notes in Reading View by default, which is required for `^[filename:L-L]` citation chips to render. Only `defaultViewMode` is written; all other keys in `app.json` are preserved. If `app.json` is malformed JSON the write is skipped and a warning is logged — the install still succeeds.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Three-tab modal. **Current States** — sortable, filterable, paginated table (dynamic row count based on modal height) of all wiki pages with their current lifecycle state and last transition timestamp; click column headers to sort; each row shows valid transition buttons with a reason dialog before committing; draft/stale badge links on the lint modal and jobs panel open the table pre-filtered to that state. **Audit Log** — full history of every state transition across all pages, searchable by slug, filterable by target state, sortable by any column, with pagination and a purge footer (keep latest N per slug, or purge events before a date). **Content Snapshots** _(v1.2.0)_ — flat list of all snapshots across all pages, grouped by slug and sorted by index when unfiltered; filter-by-slug input with a × clear button; each row shows slug, snapshot index, state transition, triggered-by, timestamp, and size; click a row to open the corresponding wiki page; **View** button shows the snapshot diff against the current file body (LCS diff, YAML frontmatter stripped); **Rollback** button restores the page body to that snapshot (saves current body first, making rollback undoable). |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports two things: dangling slugs (pages listed in ROUTING.md that no longer exist in the wiki) and unassigned slugs (pages that exist in index.md but are not listed in any ROUTING.md branch). Enabled only when ROUTING.md exists. **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n| `Graph: show knowledge graph` _(v1.1.0)_ | Open the knowledge graph panel — draggable modal with Canvas force-directed graph, type filter dropdown, cluster legend, hover tooltip showing title/slug/type/state/cluster/connections, and click-to-open page in current pane. System pages (index, overview, dashboard, purpose, log) are excluded. Capped at 300 most-connected nodes for large wikis with an explanatory banner. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Background vault monitoring _(v1.2.0)_\n\nIn addition to the command palette commands, the plugin registers a passive vault event\nlistener for automatic content snapshot capture:\n\n- **`vault.on(\"modify\")`** — fires whenever Obsidian saves a file. The handler filters to\n `.md` files at the vault root only (wiki pages, not subdirectories or scaffold files).\n- **2-second per-slug debounce** — rapid keystrokes during editing collapse into a single\n request. The timer resets on each save event and only fires after 2 seconds of quiet.\n- **Deduplication on the server** — `POST /pages/{slug}/snapshot` calls\n `AuditLog.snapshot_if_changed()`, which compares the incoming content to the last\n stored snapshot. No new row is written when content is identical.\n- **Silent failure** — if the server is offline or the slug is not in the wiki, the error\n is swallowed. The plugin never shows an error notification for background snapshot calls.\n\nThis gives rollback coverage for edits that do not trigger a lifecycle transition — for\nexample, manually fixing an orphan page, editing body copy, or correcting a citation —\nwithout requiring any explicit user action.\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [-w wiki] — generate ROUTING.md from index.md branch structure\n│ ├── validate [-w wiki] — report dangling slugs and cross-branch duplicates\n│ └── clean [-w wiki] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [-w wiki]\n├── candidates\n│ ├── list [-w wiki]\n│ ├── promote |--all [-w wiki]\n│ └── discard |--all [-w wiki]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ ├── log [slug] [-w wiki] [--state ]\n│ ├── history [-w wiki] [--index N] [--show-content] list content snapshots (newest first)\n│ └── rollback [-w wiki] [--index N] [--reason \"\"] restore page body to a snapshot\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** subscription, or want to use the free **Opencode Zen** tier, you can use either as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode Zen (free, no API key — requires connecting first: run `opencode` → `/connect` → select Zen):\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"opencode/big-pickle\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 30\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n# Citation pass (Pass 4) tuning — these two settings work together:\n# citation_source_lines — how many lines of the source the LLM sees when placing ^[...] markers.\n# Increase if lint reports out_of_range on long sources (transcripts, PDFs).\n# citation_max_tokens — output token budget for the annotated section returned by the LLM.\n# Increase if you raise citation_source_lines and have long wiki sections.\n# citation_source_lines = 400\n# citation_max_tokens = 8192\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax`, `qwen` (DashScope), and `deepseek` providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `30` | Exponential backoff base; delays are `min(base × 2^(attempt−1), 300)` seconds |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Emit `logger.warning` in server log when per-job ingest cost exceeds this threshold; job continues normally |\n| `cost.hard_gate_usd` | float | `2.00` | Permanently fail the ingest job (DEAD) and record a `cost_gate_exceeded` audit event when per-job cost exceeds this threshold |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × (backup_count + 1)` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n| `ingest.max_source_chars` | int | `32000` | Character limit applied to each source before the LLM call. Sources exceeding this limit are truncated; the page's `sources:` frontmatter entry gets `truncated: true` and lint emits a warning. Override per-run with `--max-source-chars N`. _(v1.0.0)_ |\n| `ingest.citation_source_lines` | int | `400` | Number of source lines the LLM sees during Pass 4 (citation annotation). Increase if lint reports `out_of_range` on long sources such as transcripts or PDFs. Raise `citation_max_tokens` in proportion when increasing this value. |\n| `ingest.citation_max_tokens` | int | `8192` | Output token budget for the annotated section returned by Pass 4. Increase when `citation_source_lines` is raised or wiki sections are long, to avoid truncated annotations. |\n| `query.context_wiki_pct` | float | `0.60` | Fraction of the model context window reserved for wiki source pages. _(v1.0.0)_ |\n| `query.context_history_pct` | float | `0.20` | Fraction reserved for conversation history. _(v1.0.0)_ |\n| `query.context_system_pct` | float | `0.15` | Fraction reserved for system prompt and instructions. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_index_pct` | float | `0.05` | Fraction reserved for the wiki index preamble. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_window` | int | _(auto)_ | Override the built-in context window lookup for the configured model. Use when running a local or custom model whose window size is not in the built-in table. _(v1.0.0)_ |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement is wired into the ingest pipeline: `Orchestrator._guard_ingest_cost()` is called after each ingest LLM call with the returned cost, before the job is marked complete.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | `logger.warning` in server log; job continues normally |\n| `hard_gate_usd` | $2.00 | Server/ingest path: records `cost_gate_exceeded` audit event + permanently fails job (DEAD). Interactive CLI: prompts `Proceed? [y/N]` |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `min(backoff_base_seconds × 2^(attempt − 1), 300)` \nWith the default base of 30 s: attempt 1 waits 30 s, attempt 2 waits 60 s, attempt 3 waits 120 s (all capped at 300 s). The `retry_after` timestamp is stored in the jobs table; `dequeue()` skips any job whose `retry_after` is still in the future. Applied only to transient errors (network timeouts, connection failures, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` by default. The bind address is configurable via `server.host` in `config.toml` (e.g. `host = \"0.0.0.0\"` to expose on all interfaces for LAN access). No built-in authentication — restrict via firewall when exposing beyond loopback. Remote access without a reverse proxy is not recommended on shared or networked machines.\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs and unassigned slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `-w ` / `--wiki ` to target a specific wiki.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; when all local source files are missing from disk the page is transitioned to `archived` (if only some sources are missing it is transitioned to `stale` instead). Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: lint evaluates all local (non-URL) source paths listed in the page's frontmatter. If **all** of them are missing from disk, the page is transitioned to `archived`. If only **some** are missing — meaning valid content from at least one source still exists — the page is transitioned to `stale` instead, preserving it for re-ingest rather than discarding it. _(v1.0.2: previously any single missing source triggered an immediate archive.)_\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Cascade link cleanup\n\nWhenever a page transitions to `archived` — whether triggered manually (`synthadoc lifecycle archive`), via the Obsidian Lifecycle modal, the MCP `synthadoc_lifecycle` tool, or automatically by lint detecting a missing source — Synthadoc immediately scans all other active pages and removes every `[[archived-slug]]` wikilink pointing to the now-archived page.\n\n- **Inline links** — `[[archived-slug]]` becomes plain text (the display text if an alias was used, otherwise the slug itself).\n- **List-item links** — a list item whose only content is `[[archived-slug]]` is dropped entirely.\n- **Archived pages are skipped** — pages already archived are not rewritten.\n- **System pages are skipped** — index, overview, dashboard, log, and purpose pages are excluded.\n\nThe cleanup runs synchronously before the response is returned, so the wiki is consistent immediately. The CLI and MCP responses include a `cascade_links_removed_from` field listing the affected slugs. During `lint run`, cascade is batched once after all auto-archive transitions complete, and the count is added to `dangling_links_removed` in the lint report.\n\nLint's existing periodic dangling-link cleanup (`lint run --scope orphans`) remains in place as a safety net for any dead links that predate this feature or arrive through other paths.\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc lifecycle history -w [--index ] [--show-content] [--json]\n List content snapshots captured at lifecycle transitions (newest first).\n --index inspects a single snapshot; --show-content prints the full body.\n\nsynthadoc lifecycle rollback -w --index --reason \"...\"\n Restore the page body from snapshot . The current body is saved as a new\n snapshot first, so every rollback is undoable.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n### Page Content Snapshots\n\nA content snapshot is a copy of the page body (`WikiPage.content`) stored in the\n`content_snapshot` column of a `lifecycle_events` row. Three distinct triggers produce\nsnapshots; ingest-agent transitions never do (ingest does not change existing content):\n\n| Trigger | When it fires | Dedup? |\n|---------|--------------|--------|\n| **Manual lifecycle transition** | User calls `lifecycle activate`, `lifecycle archive`, `lifecycle restore` via CLI, the Obsidian Lifecycle modal, the REST API, or the MCP tool | No — always records |\n| **Lint-driven state change** | `LintAgent._transition()` fires: draft → active (promotion), active → stale (hash mismatch), active → archived (source gone), contradicted → active (auto-resolve) | No — always records; for auto-resolve, captures the post-resolution body that was just written |\n| **Manual file edit in Obsidian** | The Obsidian plugin's `vault.on(\"modify\")` handler fires after a 2-second debounce on any `.md` file at the vault root | Yes — `AuditLog.snapshot_if_changed()` compares incoming content to the last stored snapshot and only writes a new row when content differs |\n\nFor the manual-edit trigger the `from_state` and `to_state` columns are both set to the\ncurrent page state (no lifecycle transition occurs — this is a pure content checkpoint).\n\n**Deduplication (manual edits only):** `snapshot_if_changed()` reads the last\n`content_snapshot` for the slug from the DB and compares it to the incoming content with\na string equality check. At typical wiki page sizes (< 100 KB) and the 2-second debounce\nrate, this is negligible overhead. A future optimisation may store a `content_hash`\ncolumn to avoid reading the full text from the DB; deferred until page sizes make it\nmeasurable.\n\nSnapshots are indexed 1-based, newest first. Index 1 always refers to the most recent\nsnapshot. The rollback operation:\n\n1. Records the current body as a new snapshot (making rollback undoable).\n2. Writes the target snapshot body to the `.md` file on disk.\n3. Does **not** change `page_states` or call `set_page_state`.\n\nStorage: ~3 KB per snapshot. Existing `purge_lifecycle_events()` removes rows\n(and their snapshots) on `--before` or `--keep-latest` purge.\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [...], \"input_tokens\": 1240, \"output_tokens\": 380, \"tokens_used\": 1620}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […], \"input_tokens\": N, \"output_tokens\": N, \"tokens_used\": N}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Streaming Token Audit\n\nStreaming LLM responses do not return token counts in the same way as blocking calls. To track real usage in the audit trail:\n\n**Provider-side collection** — each `LLMProvider` subclass exposes two instance attributes: `last_stream_input_tokens` and `last_stream_output_tokens`. These default to `0` and are populated from the API-specific usage field when the stream ends:\n\n| Provider | Mechanism | Verified |\n|---|---|---|\n| OpenAI | `stream_options={\"include_usage\": True}` passed to `chat.completions.create()`; final chunk carries `chunk.usage.prompt_tokens` / `completion_tokens` with empty `choices` | ✅ |\n| Anthropic | `message_start` event → `event.message.usage.input_tokens`; `message_delta` event → `event.usage.output_tokens` | ✅ |\n| Ollama | Final chunk with `done=True` → `prompt_eval_count` / `eval_count` | ✅ |\n| DeepSeek | Same `OpenAIProvider` path as OpenAI; DeepSeek's OpenAI-compatible API supports `stream_options` | ✅ |\n| MiniMax (reasoning: M2.5+) | Detects `` in stream → falls back to blocking `complete()` → captures exact counts from `resp.usage` | ✅ |\n| MiniMax (non-reasoning, e.g. M3 thinking=disabled) | Same `OpenAIProvider` path; MiniMax API silently ignores `stream_options`. Falls back to character-based estimate (÷ 3.5 chars/token) from prompt + answer lengths. Accuracy ±20%. | ✅ estimated |\n| Gemini | Same `OpenAIProvider` path via `generativelanguage.googleapis.com/v1beta/openai/`; Google's compatibility layer honours `stream_options` — verified live (Gemini 2.5 Flash Lite, 50K tokens) | ✅ |\n| Groq | Same `OpenAIProvider` path; Groq's OpenAI-compatible API supports `stream_options` | ✅ |\n| Qwen (DashScope) | Same `OpenAIProvider` path via DashScope; `stream_options` honoured — verified live (qwen-plus, 28K tokens) | ✅ |\n| Qwen (Ollama) | Ollama path → `prompt_eval_count` / `eval_count` | ✅ |\n\n**Forwarding to `done` event** — after `complete_stream()` is exhausted, `QueryAgent.run_stream()` reads the provider attributes and includes them in the final `done` event payload: `input_tokens`, `output_tokens`, `tokens_used`.\n\n**Audit record** — `Orchestrator.query_stream()` consumes the `done` event and calls `estimate_cost(model, input_tokens, output_tokens, is_local)` from `synthadoc/providers/pricing.py`, then calls `AuditDB.record_query()` with the real token count and computed cost. The audit row at `GET /audit/queries` will therefore always show non-zero values for cloud providers.\n\nWhen a provider does not report streaming usage (e.g. a future provider not yet wired), the attributes stay at `0`; the audit row records `tokens=0, cost_usd=0.0` rather than crashing, matching the pre-fix behaviour.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 5) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 5 # turns to include in each request (default: 5; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp, cascade_links_removed_from: [str]}` or `{error, cascade_links_removed_from: []}` _(cascade field added v1.0.2)_ | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions for Codex CLI / OpenCode (if present)\n├── CLAUDE.md ← same content; loaded by Claude Code (if present)\n├── GEMINI.md ← same content; loaded by Gemini CLI (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── hooks/ ← user hook scripts (always included)\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ ├── extracted/ ← text sidecars + PDF pagemaps (always included)\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `logs/`.\n\n> **v0.9.3 fix:** `hooks/` and `.synthadoc/extracted/` were previously missing from backups. Omitting `extracted/` caused the Source Viewer and Provenance citation modal to show \"Could not read source file\" after restore.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## 29. Pre-LLM Source Sanitizer\n\nEvery source document passes through a sanitizer immediately after text extraction, before any LLM call. The sanitizer removes six categories of content that could manipulate the LLM or degrade compilation quality:\n\n| Category | Action | Warning logged? |\n|---|---|---|\n| Zero-width characters (U+200B, U+200C, U+200D, U+FEFF) | Removed silently | No |\n| Bidi override characters (U+202A–U+202E, U+2066–U+2069) | Removed | Yes |\n| HTML comments (``) | Removed silently | No |\n| Hidden CSS spans (`display:none`, `visibility:hidden`) | Removed silently | No |\n| Base64 blobs ≥ 200 consecutive characters | Replaced with `[base64 content removed]` | Yes |\n| Instruction-override phrases (8 patterns: \"ignore previous instructions\", \"disregard the above\", \"override your system prompt\", etc.) | Replaced with `[redacted]` | Always |\n\nWhen content is removed and a warning is appropriate, the server logs a `WARN` entry:\n```\n[WARN] sanitizer stripped content from 'papers/survey.pdf': bidi overrides, instruction-override phrase\n```\n\nThis is a pure pre-processing step with zero LLM cost. Truncation (if configured) is applied after sanitization, so the character budget is consumed by clean text only.\n\n---\n\n## 30. Per-Source Truncation Flag\n\nLarge source documents are truncated before they reach the LLM. By default the limit is **32,000 characters** (~8,000 tokens), raised from the prior hardcoded 8,000-character limit. When a source exceeds this limit, its compiled wiki page records a `truncated: true` flag in its `sources:` frontmatter entry.\n\n### Configuration\n\n```toml\n[ingest]\nmax_source_chars = 32000 # default; raise for large PDFs or books\n```\n\n### Per-ingest override\n\nThe limit can be raised for a single ingest run without modifying the config:\n\n```bash\nsynthadoc ingest papers/large-textbook.pdf --max-source-chars 128000\n```\n\nThe same `max_source_chars` field is accepted by the HTTP `POST /jobs/ingest` body and the MCP `synthadoc_ingest` tool.\n\n### Lint surfacing\n\n`synthadoc lint` emits a warning for any page that has a truncated source:\n\n```\n[WARN] quantum-computing.md: source 'papers/large-survey.pdf' was truncated at ingest\n (source exceeded max_source_chars=32000 — 87,412 chars in source).\n To re-ingest with a higher limit (this source only):\n synthadoc ingest papers/large-survey.pdf --max-source-chars 128000\n To raise the limit for all future ingests:\n set [ingest] max_source_chars = 128000 in your config\n```\n\n---\n\n## 31. Proportional Context Budget\n\nQuery context is allocated proportionally to the configured model's context window, replacing the prior hard cap. The default allocation:\n\n| Slice | Default | Purpose |\n|---|---|---|\n| `context_wiki_pct` | 60% | Wiki page content (ranked by BM25 + vector score) |\n| `context_history_pct` | 20% | Chat turn history (newest-first, oldest dropped when over budget) |\n| `context_system_pct` | 15% | System prompt and purpose document |\n| `context_index_pct` | 5% | `ROUTING.md` and search index |\n\nPercentages must sum to ≤ 100. Configure in `[query]`:\n\n```toml\n[query]\ncontext_window = 0 # 0 = auto-detect from known model map\ncontext_wiki_pct = 60\ncontext_history_pct = 20\ncontext_system_pct = 15\ncontext_index_pct = 5\n```\n\n> **Note:** `context_system_pct` and `context_index_pct` are parsed and validated but not yet enforced as hard caps in this release — the system prompt uses a conservative fixed limit, which is well within the configured 15% slice for all supported models. Full per-slice enforcement is planned for v1.1.\n\n### Model context window map\n\nSynthadoc ships a built-in prefix-matched table:\n\n| Model | Context window |\n|---|---|\n| `claude-opus-4*`, `claude-sonnet-4*`, `claude-haiku-4*` | 200,000 tokens |\n| `gpt-4o*`, `gpt-4-turbo*` | 128,000 tokens |\n| `gpt-4` (exact) | 8,192 tokens |\n| `gpt-3.5-turbo*` | 16,385 tokens |\n| Unknown / fallback | 128,000 tokens |\n\nSet `context_window = N` in config to override the map (useful for local Ollama models with non-standard context sizes).\n\nAt the defaults on a 200k-token model: wiki slice ≈ 480,000 chars — far more than any practical query needs. On constrained models, the budget scales down automatically so the same config works across model families.\n\n---\n\n## 32. Knowledge Graph\n\nSynthadoc computes a wikilink graph across all active and draft pages during every lint run. The graph is stored in two tables in `audit.db`:\n\n```sql\nCREATE TABLE graph_nodes (\n slug TEXT PRIMARY KEY,\n cluster_id INTEGER NOT NULL,\n updated_at TEXT NOT NULL\n);\n\nCREATE TABLE graph_edges (\n from_slug TEXT NOT NULL,\n to_slug TEXT NOT NULL,\n weight INTEGER NOT NULL DEFAULT 1,\n edge_type TEXT NOT NULL DEFAULT 'mixed',\n PRIMARY KEY (from_slug, to_slug)\n);\n```\n\nEdge weight is a composite signal from two sources:\n\n| Edge type | Source | Weight contribution |\n|-----------|--------|---------------------|\n| `wikilink` | `[[slug]]` occurrences in page body | +1 per occurrence |\n| `co_source` | pages compiled from the same source file (matched by hash) | +2 per shared source |\n\nMost edges are `mixed` (have both types). Pure `co_source` edges surface hidden relationships — pages compiled from the same source document are linked immediately after ingest, before any wikilinks are created. `edge_type` is stored on each edge so the web UI can render co-source edges with a different visual style.\n\nCluster IDs are assigned by Louvain community detection — pages that link densely to each other end up in the same cluster.\n\n### REST API\n\n**`GET /graph`** — returns the full graph with per-node enrichment:\n\n```json\n{\n \"status\": \"ready\",\n \"node_count\": 45,\n \"edge_count\": 123,\n \"cluster_count\": 4,\n \"nodes\": [\n {\"slug\": \"quantum-error-correction\", \"title\": \"Quantum Error Correction\",\n \"type\": \"concept\", \"state\": \"active\", \"cluster_id\": 2}\n ],\n \"edges\": [\n {\"from\": \"quantum-error-correction\", \"to\": \"shor-algorithm\", \"weight\": 3, \"edge_type\": \"mixed\"}\n ]\n}\n```\n\nWhen the graph tables are empty (first call after upgrade, or before the first lint run), the server returns `{\"status\": \"computing\"}` and starts a background build immediately. The web UI polls every 2 seconds and renders the graph when the status becomes `\"ready\"`.\n\n### Web UI\n\nThe **Graph** tab (alongside Chat in the top nav) renders a D3.js force-directed graph. Nodes are colored by cluster. Edge thickness is proportional to weight (1–4 px). Pure `co_source` edges (no wikilink) are rendered with a dashed stroke, making hidden source-based relationships visually distinct from explicit cross-links. Clicking a node opens a sidebar showing the page title, type and lifecycle state badges, cluster assignment, and an **\"Ask about this →\"** button that switches to the Chat tab with a pre-filled query.\n\nControls: zoom in/out (scroll or pinch), drag nodes, filter by page type.\n\n### System page exclusion _(v1.1.0)_\n\nThe five structural system pages — `overview`, `index`, `dashboard`, `purpose`, and `log` — are excluded from both `graph_nodes` and `graph_edges`. These pages link to nearly every other page and would act as false hubs, inflating edge counts and distorting Louvain cluster detection. System page exclusion is applied inside `_build_graph()` before any edge computation.\n\n### Obsidian knowledge graph panel _(v1.1.0)_\n\n→ Full description: [§33 Multi-Platform Agent Skill Files](#33-multi-platform-agent-skill-files) covers scaffold; the graph panel itself is documented in [§8 Obsidian Plugin](#8-obsidian-plugin) command table.\n\nThe Obsidian plugin includes a **draggable modal** knowledge graph panel (`Graph: show knowledge graph` command). It fetches from the same `GET /graph` endpoint as the web UI and renders a force-directed Canvas 2D simulation using Verlet integration (no external library). System pages are already excluded from the server response.\n\n| Feature | Obsidian panel | Web UI Graph tab |\n|---------|----------------|-----------------|\n| Engine | Canvas 2D + Verlet | D3.js SVG |\n| Node click | Opens the wiki page in the current pane | Opens the node sidebar with \"Ask about this →\" |\n| **Best for** | **Navigating to and editing pages** | **Asking questions about a page** |\n| Draggable panel | Yes — drag header to reposition | Fixed tab layout |\n| Node cap | 300 most-connected (see note) | No cap |\n| Cluster legend | Yes | Yes |\n| Type filter | Yes | Yes |\n\n**Rule of thumb:** Use the **web UI** when you want to ask questions about a node. Use the **Obsidian panel** when you want to navigate to and edit a page.\n\n**Node cap rationale:** The Obsidian panel uses a hand-written Verlet integrator with O(n²) pairwise charge repulsion — every node repels every other node on each animation tick. At 300 nodes that is ~90,000 operations per frame, which runs smoothly at 60 fps inside Obsidian's Electron renderer. At 500 nodes it is ~250,000 operations per frame and starts to lag noticeably. The web UI has no cap because D3's `forceManyBody` uses a Barnes-Hut tree approximation (O(n log n)), which scales to thousands of nodes without frame-rate issues. When the cap is hit, nodes are ranked by degree and the 300 most-connected are shown; a banner reports the total page count and the number of hidden nodes, and suggests using the type filter to bring the set below 300.\n\n→ User walkthrough: [Quick-Start Guide §24 — Knowledge Graph](docs/user-quick-start-guide.md#knowledge-graph)\n\n---\n\n## 33. Multi-Platform Agent Skill Files\n\n\n\n`synthadoc init` and `synthadoc scaffold` write three agent skill files at the wiki root:\n\n| File | Format | Read by |\n|------|--------|---------|\n| `AGENTS.md` | OpenAI Agents SDK / Codex / OpenCode | Codex CLI, OpenCode, generic agents |\n| `CLAUDE.md` | Claude Code / Anthropic | Claude Code (loaded automatically when the wiki folder is opened) |\n| `GEMINI.md` | Gemini CLI | Gemini CLI |\n\nAll three files carry **identical content**: a title block, the LLM-generated domain guidelines for the wiki, a quick-reference CLI command table, and an MCP tool table. The only difference is the heading comment. This means any AI coding tool that opens the wiki root gets full Synthadoc guidance without manual setup.\n\n### Content generated by `scaffold`\n\nEach skill file is generated from the same LLM call that produces `purpose.md` and `index.md`. The LLM receives the current wiki page list and domain description, then generates:\n\n1. **Domain label** — a precise human-readable name inferred from wiki content (e.g. `History of Computing`, not `My Wiki`)\n2. **Domain guidelines** — 3 domain-specific ingest and query guidelines as plain sentences\n3. **Quick Reference table** — pre-filled with the current wiki name and port\n\nIf the wiki was created before v1.1.0, running `synthadoc scaffold` once regenerates all three files and adds `CLAUDE.md` and `GEMINI.md` to any wiki that only had `AGENTS.md`.\n\n### Template source\n\nThe instruction body is defined in `synthadoc/cli/_init.py` as `_AGENT_INSTRUCTION_BODY` and shared by all three templates, keeping the files permanently in sync. The scaffold agent substitutes `{domain}`, `{guidelines}`, and `{port}` at write time.\n\n---\n\n## 34. Synthadoc Scaffold Marker\n\n\n\nThe scaffold marker `` (written as an HTML comment) separates **user-authored content** (preserved) from **scaffold-managed content** (rewritten on each `synthadoc scaffold` run).\n\n### Single-marker mode (index.md)\n\nThe marker appears once, below the H1 heading in `index.md`. Everything above the marker is the user zone; everything below is replaced with new LLM-generated categories.\n\n```markdown\nMy custom introduction — maintained by hand.\n\n\n\n## Pioneers and Visionaries\n- [[alan-turing]]\n...\n```\n\nIf the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### Per-section multi-marker mode (purpose.md)\n\n`purpose.md` uses a **per-section** approach: the marker appears inside each `## Section` heading. Within a section, content the user writes **above** the marker is preserved; content below is replaced with fresh LLM output. Sections without a marker in the existing file are replaced entirely (and a marker is added). Sections in the existing file that the scaffold template does not include are kept unchanged.\n\n```markdown\n## Overview\n\nMy custom overview note.\n\n\n\nLLM-generated overview content goes here.\n\n## What Belongs\n\n\n\nLLM-generated scope definition.\n```\n\nThis design lets users annotate any section of `purpose.md` without losing their notes on the next scaffold run.\n\n### Marker preservation guarantee\n\n- `preserve_user_zone()` in `scaffold_agent.py` handles both modes.\n- If a section has no marker in the existing file, `preserve_user_zone()` treats the entire section body as user content and preserves it, replacing only what falls below any marker it finds.\n- The scaffold NEVER touches `config.toml`, `dashboard.md`, or any wiki page (`wiki/*.md`).\n\n→ User walkthrough: [Quick-Start Guide §14 — Enrich the wiki with scaffold](docs/user-quick-start-guide.md#scaffold)\n\n---\n\n## Customization\n\nSynthadoc exposes four extension points — all hot-loaded, no server restart required.\n\n### Custom skills (new file formats)\n\nSubclass `BaseSkill` (Apache-2.0 — no AGPL obligation on your skill code), drop the folder in `/skills/` or `~/.synthadoc/skills/`, and Synthadoc picks it up on the next ingest. Skills match by file extension or intent prefix and support any Unicode text, including CJK prefixes.\n\n→ Full interface, manifest format, and examples: [§5 Skills System](#5-skills-system) and [§17 Plugin Development Guide](#17-plugin-development-guide)\n\n### Custom LLM providers\n\nSubclass `LLMProvider` from `synthadoc/providers/base.py` (Apache-2.0) and place the file in `~/.synthadoc/providers/` or the wiki `providers/` directory. Switch active provider with one config line.\n\n→ Full interface and wiring instructions: [§10 Configuration — Provider switching](#10-configuration)\n\n### Hooks\n\nShell scripts (any language) that fire on `on_ingest_complete` and `on_lint_complete`. Receive a JSON context on stdin. Set `blocking = true` to gate the operation on the hook's exit code — useful for CI pipelines and quality gates.\n\n→ Full event schema, context JSON, and blocking behaviour: [§11 Hook System](#11-hook-system)\n\n### Cache control\n\nThree cache layers (embedding, LLM response, provider prompt cache) invalidate automatically on source-file change (SHA-256). Force a fresh call with `--force`, or wipe all cached responses with `synthadoc cache clear -w `.\n\n→ Layer-by-layer breakdown and invalidation rules: [§12 Cache System](#12-cache-system)\n\n### Multi-Platform Agent Skill Files\n\nEvery Synthadoc wiki ships three companion files that give AI coding agents a complete, self-contained operating guide for the wiki:\n\n| File | Platform |\n|---|---|\n| `AGENTS.md` | Codex CLI, OpenCode, and any agent that reads the AGENTS.md convention |\n| `CLAUDE.md` | Claude Code (this session's tool) — highest-priority instruction source in Claude's hierarchy |\n| `GEMINI.md` | Gemini CLI — relevant because Synthadoc's default LLM provider is `gemini-2.5-flash-lite` |\n\nAll three files share identical body content generated from the same template; they differ only in their H1 heading so each agent recognises its own file first. The body covers:\n\n- **Domain guidelines** — LLM-generated bullet list of domain-specific ingest and query rules (produced by `scaffold`)\n- **Quick reference table** — every common CLI command with the correct `synthadoc` syntax\n- **Server startup** — how to start the server and verify it is running\n- **Ingest examples** — local file, URL, YouTube, agent session `.jsonl`, force reingest, dry-run\n- **Query** — basic and streaming query syntax; citation marker format\n- **Lint** — when to run and what each check does\n- **Lifecycle** — transition commands with state names\n- **Page schema** — frontmatter structure with all fields\n- **MCP tools table** — all 12 tools with purpose descriptions\n\n**How the files are created and updated:**\n\n1. `synthadoc init` — writes all three files with default domain guidelines and the wiki's configured port.\n2. `synthadoc scaffold` — regenerates all three with LLM-produced domain guidelines derived from the current wiki state. The `scaffold` command prints a confirmation line for each file.\n\n**Use cases:**\n\n- Open Claude Code in a Synthadoc wiki directory and it immediately knows how to ingest, query, lint, and manage lifecycle — no additional setup. Claude Code can also start the server in background mode (`synthadoc serve -w -b`), stop it cleanly, and orchestrate multi-step workflows, effectively acting as a control plane for the wiki.\n- Gemini CLI reads `GEMINI.md` and can run `synthadoc` commands with the correct flags without asking the user for help.\n- CI pipelines with Codex CLI use `AGENTS.md` to drive automated lint and lifecycle transitions on PR merge.\n- Agent session files produced by Claude Code (`.jsonl` under `~/.claude/projects/`) can themselves be ingested into the wiki, capturing the reasoning behind design decisions alongside the source documents.\n\n→ See [README.md — Interfaces & Integration](../README.md#interfaces--integration) for the feature comparison row and quick-start instructions.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v1.1.2\n\n- **Exponential backoff on transient URL fetch failures** — jobs failing with transient errors (connection timeouts, 5xx responses, dropped TCP connections including `RemoteProtocolError`) now re-queue with exponential backoff: `min(base × 2^(attempt−1), 300)` seconds (default base 30 s). Previously all failures retried immediately, exhausting the retry budget within seconds. `dequeue()` skips any job whose `retry_after` timestamp is still in the future; other jobs run normally during the wait window. See [§14 Job Queue](#14-job-queue).\n- **Accurate retry outcome in job warning logs** — the transient-failure warning now reads the current attempt count from the database before logging, so it correctly reports `will retry` vs. `final attempt` (was always `will retry` regardless of retry state).\n- **Scheduler subprocess timeout** — scheduled jobs (`lint`, `ingest --batch`, etc.) are now guarded by `asyncio.wait_for(proc.communicate(), timeout=job_timeout_seconds)`; a hung subprocess is killed after the timeout and the run is recorded as `failed`. Previously a stalled subprocess could block the scheduler task indefinitely. `job_timeout_seconds` defaults to 600 s and is now shown (uncommented) in fresh-init `config.toml`.\n- **Streaming query token and cost accuracy** — the streaming query audit record was written with `tokens=0, cost_usd=0.0`; it now records the actual LLM token usage and cost from the streamed response.\n- **Cost hard gate timing** — the cost hard gate now fires before LLM tokens are spent and pages are written; previously the gate ran after the LLM call completed, allowing partial page writes before rejection.\n- **Session state memory growth bounded** — `_session_state` in the HTTP server is pruned at 100 concurrent sessions (LRU eviction) to prevent unbounded memory growth on long-running servers.\n- **Streamed `` block cleanup** — a leaked fallback asyncio task is now cancelled when a streaming response ends inside an unclosed `` block (e.g. reasoning-model responses cut off mid-think).\n- **Config validation for staging fields** — `staging_policy` values other than `off`, `all`, and `threshold`, and non-numeric `staging_confidence_min`, were silently accepted and caused runtime errors; they now raise a `ConfigError` at server startup.\n- **Job status guard on retry** — retrying a job that is already `in_progress`, `done`, or `skipped` now returns an error instead of silently resetting it to `pending`.\n- **Jobs endpoint performance** — `GET /jobs/{id}` and `DELETE /jobs/{id}` now use a direct primary-key lookup instead of a full table scan; eliminates O(N) reads on large job queues.\n- **Vector migration GC safety** — the background vector migration `asyncio.Task` is now held in a strong reference to prevent premature garbage collection mid-run.\n- **Quota error stack trace suppression** — Qwen 403 errors and Claude Code session-limit exhaustion now emit a concise warning naming the actionable config key, instead of a full stack trace.\n- **Scaffold agent JSON robustness** — `MiniMax-M3` and other reasoning models can truncate mid-JSON; the scaffold agent now handles truncated JSON output gracefully rather than failing the job.\n- **Staging/Candidates section in agent skill files** — `AGENTS.md`, `CLAUDE.md`, and `GEMINI.md` generated by `synthadoc scaffold` now include a `## Staging and Candidates` section.\n- **Citation failure pagination fix** — `GET /audit/citation-failures` was applying the `LIMIT` in SQL before filtering by `page_slug`, returning incorrect results for paginated per-page queries; filtering now runs before the limit.\n\n### v1.1.0\n\n- **Weighted knowledge graph edges** — graph edges now carry two signals: wikilink occurrences (+1 each) and co-source connections — pages compiled from the same source file (matched by SHA-256 hash) — (+2 per shared source). `edge_type` field added to `graph_edges` table (`wikilink`, `co_source`, or `mixed`). `GET /graph` exposes `edge_type` per edge. Web UI renders edge thickness proportional to weight (1–4 px) and dashed lines for pure co-source edges. Co-source edges appear immediately after ingest — before any wikilinks — surfacing hidden relationships in the graph. Schema migrated to version 3.\n- **Multi-platform agent skill files** — `synthadoc init` and `synthadoc scaffold` write `CLAUDE.md`, `AGENTS.md`, and `GEMINI.md` at wiki root; all three carry LLM-generated domain guidelines and the full CLI quick-reference so Claude Code, Codex CLI, OpenCode, Pi, and Gemini CLI users get first-class guidance without manual setup. Run `synthadoc scaffold` once on any pre-v1.1.0 wiki to generate the new files. See [§33](#33-multi-platform-agent-skill-files).\n- **Session history ingestion skill** — `.jsonl` session files from Claude Code and Codex CLI are ingested as wiki pages; human turns and substantive assistant responses extracted; tool calls, thinking blocks, and sub-agent scaffolding skipped; format auto-detected; `suggested_slug` derived from session date and first user message. Converts ephemeral problem-solving knowledge into searchable wiki entries. See [§5 Skills System — Session Skill](#session-skill--ai-session-history-ingestion).\n- **Obsidian knowledge graph panel** — new `Graph: show knowledge graph` command in the Obsidian plugin opens a draggable modal with a Canvas 2D force-directed graph. Features: Verlet integration, type filter dropdown, cluster legend, hover tooltip (title, slug, type, state, cluster, connections), click-to-open page in current pane, node drag to repin. Closes only via the × button — Escape and outside-click do not close the panel, matching the behaviour of other Synthadoc modals. **Large-wiki cap:** when the node set exceeds 300 (after any type filter is applied), nodes are ranked by degree (total edge count) and the 300 most-connected are displayed; edges to capped-out nodes are removed; a banner explains the cap and suggests using the type filter to narrow the view below 300. See [§8 Obsidian Plugin](#8-obsidian-plugin) and [§32 Knowledge Graph — Obsidian panel](#obsidian-knowledge-graph-panel-v110).\n- **Per-section scaffold markers in `purpose.md`** — `purpose.md` now uses per-section `` markers. Users can write content above the marker in any `## Section` block; that content is preserved on every re-scaffold. A single-marker mode for `index.md` continues to work as before. See [§34 Synthadoc Scaffold Marker](#34-synthadoc-scaffold-marker).\n- **System pages excluded from knowledge graph** — `overview`, `index`, `dashboard`, `purpose`, and `log` are filtered out of `_build_graph()` before any edge computation so they cannot act as false hubs in Louvain cluster detection. Both the Obsidian graph panel and the web UI Graph tab reflect this exclusion.\n- **Web UI graph cluster legend** — the web UI Graph tab now shows a cluster legend below the canvas (matching the Obsidian plugin panel), with a color swatch and label for each Louvain cluster present in the filtered node set.\n- **`_merge_section_into_page` deduplication** — force re-ingest of the same source file (e.g. an updated `.jsonl` session) now replaces the page body rather than appending a duplicate section. `_files_match()` normalizes absolute vs. relative paths so the same file is recognized regardless of how it was passed to the ingest command.\n- **`purpose.md` domain label inference** — `scaffold` now infers a precise `domain_label` from current wiki content (e.g. `History of Computing`) instead of using a generic placeholder; the label is used in all three agent skill files and in `purpose.md`'s Overview section.\n- **CI: Windows runner fix** — `scripts/sync_plugin.py` and `scripts/sync_web_ui.py` now use `shutil.copy2` instead of `cp -f`, resolving CI failures on Windows runners for the sync check.\n\n### v1.0.2\n\n- **Cascade link cleanup on archive** — archiving any page immediately removes every `[[slug]]` wikilink pointing to it from all other active pages, without waiting for the next `lint run`. Inline links become plain text; list-item-only links are dropped entirely. The HTTP `POST /lifecycle/transition` and MCP `synthadoc_lifecycle` responses now include a `cascade_links_removed_from` field listing the rewritten page slugs. During `lint run`, cascade runs once after all auto-archive transitions complete and the count is included in the lint report's `dangling_links_removed` field. System pages (index, dashboard, purpose, log, overview) and already-archived pages are skipped. The existing dangling-link lint pass remains as a safety net for dead links that predate this release.\n- **Multi-source partial-archive fix** — when a page has multiple local source files and only some are missing from disk, lint now transitions the page to `stale` (not `archived`), preserving it for re-ingest. The page is only archived when every local source file is gone. URL sources are not affected by this check.\n\n### v1.0.1\n\n- **Scaffold JSON extraction hardened** — replaced the greedy regex used to extract the LLM's JSON response with a brace-balanced parser; eliminates false-positive matches on `[[wikilink]]` syntax that previously caused `json.loads()` failures and silent scaffold corruption\n- **Generated file validation** — scaffold now validates every LLM-generated file (index, purpose, AGENTS.md) for minimum length, required frontmatter, and `[[wikilink]]` presence before writing to disk; validation failure marks the job as failed with a structured error message rather than silently writing a malformed page\n- **ROUTING.md regeneration validation** — `routing init` and scheduled routing regeneration now validate the regenerated file for minimum line count and anchor presence before applying it; prevents a corrupt or truncated ROUTING.md from entering the wiki\n- **Scaffold failure reporting** — scaffold skip and failure reasons are now surfaced in the terminal output and job error field; previously silent failures (e.g. LLM timeout, JSON parse error) are now visible without tailing the JSON log\n- **Install-time scaffold removed** — `synthadoc install` no longer runs the LLM scaffold call during installation; scaffold is a separate step (`synthadoc scaffold`) that the user runs after setting their API key, avoiding confusing API-key-not-set errors at install time\n- **AquaFlow LLM evaluation** — 15-question M&A due-diligence benchmark across five models (MiniMax-Think M3, Claude Opus 4.8, Claude Sonnet 4.6, DeepSeek-R1, Qwen Plus); full report at `docs/example/aquaflow/evaluation/report/llm-query-benchmark.md`\n\n### v1.0.0\n\n- **Pre-LLM source sanitizer** — every source document passes through a sanitizer before any LLM call; removes six categories of potentially harmful content (zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars, instruction-override phrases); a `WARN` log entry is emitted when bidi overrides or instruction-override phrases are stripped; zero LLM cost; applied before truncation so the character budget is consumed by clean text only\n- **Per-source truncation flag** — default character limit raised from 8,000 to 32,000 characters per source (~8,000 tokens); truncated sources are flagged with `truncated: true` in the page's `sources:` frontmatter; `--max-source-chars N` CLI flag and matching `POST /jobs/ingest` body field and MCP `synthadoc_ingest` parameter override the limit per run; `synthadoc lint` emits a warning for any page with a truncated source and suggests the override command; configurable via `[ingest] max_source_chars` in `config.toml`\n- **Proportional context budget** — query context is allocated proportionally to the configured model's context window; four configurable slices: `context_wiki_pct` (60%), `context_history_pct` (20%), `context_system_pct` (15%), `context_index_pct` (5%); built-in prefix-matched model context window table covers Claude 4, GPT-4o/turbo/3.5, and GPT-4; unknown models fall back to 128,000 tokens; `context_window = N` in `[query]` overrides the table for local or custom models\n- **Knowledge graph** — wikilink graph computed across all active and draft pages during every lint run; stored in `graph_nodes` and `graph_edges` tables in `audit.db`; edge weight = number of `[[wikilink]]` references between two pages; cluster IDs assigned by Louvain community detection; `GET /graph` REST endpoint returns nodes with per-node enrichment (title, type, state, cluster) and edges; first call after upgrade triggers a background build and returns `{\"status\":\"computing\"}`; web UI **Graph** tab (D3.js force-directed graph) — nodes colored by cluster, click to see page details and an \"Ask about this →\" button that opens a pre-filled chat query\n- **Citation accuracy & rendering improvements** — six correctness fixes to the ingest citation pipeline: (1) full source text passed to the decision LLM so citations can reference any line in the document; (2) `_annotate_citations` bugs A–F resolved (false-positive matches, off-by-one line ranges, duplicate markers, escaped bracket handling, multi-paragraph span errors, and source filename normalisation); (3) `_has_citations` check added — lint warns and suggests a model upgrade when a page has zero citation markers after synthesis; (4) `Key Data` section extraction hardened so numerical facts, rates, and formulas are preserved verbatim across the annotation pass; (5) `obsidianCitationsToGfm` rendering fixed in the web UI — citation superscripts now display correctly in the chat panel alongside streamed answers; (6) reading view set as the default Obsidian display mode for wiki pages\n\n### v0.9.3\n\n- **Backup now includes `hooks/` and `.synthadoc/extracted/`** — both directories were silently omitted from backups. `hooks/` contains user-customised hook scripts; `.synthadoc/extracted/` contains per-source text sidecars and PDF pagemaps written by the ingest agent. Without these, the Source Viewer and Provenance citation modal showed \"Could not read source file\" after restore.\n- **PyPI publish workflow** — `publish.yml` triggers on GitHub Release, builds the wheel with hatchling, and uploads to PyPI via OIDC trusted publishing (no API token stored). `pip install synthadoc` now works.\n- **Web UI bundled in pip package** — `synthadoc/data/web-ui/dist/` is now included in the wheel. `synthadoc web` previously returned 503 on pip-installed instances because the React dist was not packaged.\n- **CI sync checks** — PRs now fail if `synthadoc/data/obsidian-plugin/`, `synthadoc/data/web-ui/`, or `README-pypi.md` are out of sync with their source. All sync-check steps use `shell: bash` for Windows runner compatibility.\n- **`bump_version.py` extended** — now also updates `synthadoc/data/obsidian-plugin/manifest.json` alongside the source manifest, preventing version drift between the bundled copy and the plugin source.\n\n### v0.9.2\n\n- **`synthadoc/data/web-ui/dist/` packaged** — web UI dist files committed to the repo and bundled in the wheel via hatchling artifacts.\n- **`scripts/sync_web_ui.py`** — helper script to sync `web-ui/dist/` into `synthadoc/data/web-ui/dist/` after `npm run build`.\n\n### v0.9.1\n\n- **PyPI metadata** — added `readme`, `keywords`, `classifiers`, and `[project.urls]` to `pyproject.toml` for correct PyPI rendering.\n- **`README-pypi.md`** — generated from `README.md` by `scripts/prepare_pypi_readme.py`; strips `pypi-strip` blocks and rewrites relative links to absolute GitHub URLs.\n- **Branch protection compatibility** — removed post-merge `build-plugin` CI job that pushed compiled plugin directly to `main`. Plugin sync is now enforced in PRs: CI fails if `synthadoc/data/obsidian-plugin/` is out of sync.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 5). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log/history/rollback`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n"}, "71558f9fb29fe2a87c2917d87766575684228185": {"t": 1785767239.5314305, "n": 229186, "h": "783794f6497010c7", "c": "# Synthadoc — Design Document\n\n**Version:** 1.1.3 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n29. [Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer)\n30. [Per-Source Truncation Flag](#30-per-source-truncation-flag)\n31. [Proportional Context Budget](#31-proportional-context-budget)\n32. [Knowledge Graph](#32-knowledge-graph)\n33. [Multi-Platform Agent Skill Files](#33-multi-platform-agent-skill-files)\n34. [Synthadoc Scaffold Marker](#34-synthadoc-scaffold-marker)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for Codex CLI, OpenCode, and generic OpenAI agents\n CLAUDE.md ← same content as AGENTS.md; loaded automatically by Claude Code\n GEMINI.md ← same content as AGENTS.md; loaded automatically by Gemini CLI\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the full source text (bounded by `max_source_chars`) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `flag` / `skip`). Pages with `status='active'` are protected — sources that conflict with them trigger `flag` rather than `update` (RULE 1b).\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Orchestrator calls `_guard_ingest_cost()` on the returned cost: soft warn → `logger.warning` only; hard gate → `cost_gate_exceeded` audit event + `fail_permanent` (job → DEAD). On success, job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 0b — Key Data Extraction | None (regex) | Deterministic pre-processor: extracts numbers, percentages, currency amounts, ratios, formulas, underscore identifiers, and date ranges from source text via regex; appends them as a `[Key Data — extracted by pre-processor]` section before any LLM sees the text. Zero token cost; cannot hallucinate. Anchors the synthesis LLM to all quantitative candidates. |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads the full source text (bounded by `max_source_chars`) + BM25 candidates + `purpose.md` scope + status of each candidate page. Outputs per-page action: `create`, `update`, `flag`, `skip`. **RULE 1b:** a page with `status='active'` is human-reviewed and authoritative — when the source provides a conflicting value, date, formula, or conclusion, the action must be `flag`, not `update`. The only exception is a source that adds a topic the page does not yet mention at all. Decision cache version: `v2`. |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. If no citation markers are present in the returned body, a `citation_pass4_no_markers` WARNING audit event is written — the page is still saved but the condition is flagged for re-ingest with a more capable model. Zero-citation results are not cached so a subsequent re-ingest can succeed. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic. **Suppressed when TF fallback was used** (a small corpus with broad coverage may legitimately return fewer than 3 candidates — that is expected, not a gap).\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n| Citation presence (Check 5b) | Page body has ≥ 50 words and zero `^[filename:L-L]` citation markers → WARNING. Diagnostic only — does not block promotion to `active`. Indicates the annotation pass failed to produce markers, usually a model-compatibility issue (use Gemini 2.5 Flash or higher for reliable citation annotation). |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n### ActionAgent\n\nDispatches action-intent queries from the chat UI (e.g. \"activate a draft page\", \"show lint report\", \"archive a stale page\", \"run scaffold\"). Parses the user's free-text intent into a structured action via an LLM extraction prompt, then executes the action against the wiki and returns a human-readable result. When the intent is ambiguous (e.g. \"activate a page\" without specifying which), sets `needs_clarification=True` and returns a prompt and candidate list for the web UI to render as chip buttons.\n\n### ScaffoldAgent\n\nGenerates and updates the wiki's structural pages. Reads all current wiki pages and asks the LLM to organise them into 5–8 domain-appropriate categories, then writes `wiki/index.md` (category index with wikilinks), `wiki/purpose.md` (scope statement), and `AGENTS.md` (ingest and query guidelines). Also stamps a `categories:` field on each page's YAML frontmatter. When `ROUTING.md` already exists, regenerates it to stay in sync with the updated index structure.\n\n### RewriteAgent\n\nRewrites follow-up questions in multi-turn conversations into self-contained, standalone form before BM25 retrieval. Converts context-dependent phrases (\"What came after that?\", \"Tell me more about his early life\") into explicit questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the correct pages without relying on conversation context. Only invoked when conversation history is non-empty. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### SummarizeAgent\n\nCompresses the oldest conversation turns into a single `[Session summary: …]` assistant turn when a session exceeds `conversation_history_turns`. Prevents unbounded context growth across long sessions. Emits a `notice` SSE event the first time compression occurs so the user can see that earlier context was condensed. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### ContextAgent\n\nBuilds token-budgeted context packs for MCP tool consumers. See [Section 20 — Context Packs](#20-context-packs) for full detail.\n\n### SearchDecomposeAgent\n\nDecomposes a web search intent into 1–4 terse keyword search strings optimised for authoritative source retrieval. Uses a separate prompt from `QueryAgent`'s question decomposition — search decomposition asks \"what distinct search strings would find the best sources?\" (terse keyword phrases) rather than \"what distinct questions does this ask?\" (natural-language sub-questions). See [Web search fan-out](#web-search-fan-out) in the IngestAgent section.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n| `session` | `.jsonl` | `claude session`, `codex session`, `cursor session`, `ai session`, `session history` | Extracts human-readable turns from AI coding session transcripts. Supports Claude Code JSONL format and Codex/Cursor format. Filters tool calls, thinking blocks, and sub-agent scaffolding. Short turns (< 20 assistant words, < 3 user words) are skipped. No external dependencies. |\n\n### Session Skill — AI Session History Ingestion\n\nThe `session` skill turns AI coding session history files (`.jsonl`) into searchable wiki pages capturing the problem-solving knowledge exchanged during the session.\n\n**Supported formats**\n\n| Format | File origin | Detection |\n|--------|-------------|-----------|\n| Claude Code | `~/.claude/projects//.jsonl` | `obj.type` ∈ `{\"user\", \"assistant\"}` with nested `obj.message` |\n| Codex / Cursor | Export from OpenAI Codex or Cursor IDE | `obj.role` at top level, no `message` wrapper |\n\nFormat is auto-detected from the first 30 parseable lines. Files that match neither format are parsed as Codex (fallback).\n\n**Filtering rules**\n\n| Content | Kept? | Reason |\n|---------|-------|--------|\n| User text ≥ 3 words | ✓ | Substantive question or instruction |\n| Assistant text ≥ 20 words | ✓ | Substantive answer |\n| User text < 3 words (`\"ok\"`, `\"yes\"`) | ✗ | Too terse to be useful |\n| Assistant text < 20 words | ✗ | Acknowledgement or one-liner |\n| `isSidechain: true` lines | ✗ | Sub-agent scaffolding (not the user's conversation) |\n| `tool_use` blocks | ✗ | Shell commands / file writes — not final output |\n| `tool_result` blocks | ✗ | Raw output — avoids leaking file contents or credentials |\n| `thinking` blocks | ✗ | Internal reasoning — not the final answer |\n| `permission-mode`, `system`, `last-prompt` lines | ✗ | Session metadata, not conversation |\n\nAfter extraction the text passes through Synthadoc's standard pre-LLM source sanitizer\n(zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars,\ninstruction-override phrases) — the same step applied to every PDF, DOCX, URL, and other source type.\nSee [§29 Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer).\n\n**Output format**\n\nEach kept turn is labelled and separated by `---`:\n\n```\n[USER]\nHow do I implement a sliding window algorithm in Python?\n\n---\n\n[ASSISTANT]\nA sliding window algorithm maintains a contiguous subarray by advancing two\npointers simultaneously…\n```\n\n**`suggested_slug`**\n\nThe skill sets `metadata[\"suggested_slug\"]` to `session-YYYY-MM-DD-` where the date comes from the file's `mtime` and the topic from the first substantive user turn (first 6 words, slugified). Example: `session-2026-07-15-how-do-i-implement-sliding`.\n\n**Limitations**\n\n- No chunking in v1.1 — very long sessions are truncated at `max_source_chars` (default 400 000 chars). Split large archives into individual `.jsonl` files.\n- Tool output is excluded by design — re-ingest the original source files if you need the file contents in the wiki.\n- Re-ingesting the same `.jsonl` file uses Synthadoc's standard source-hash dedup — no duplicate pages are created.\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n| `content_snapshot` | TEXT | Full page body at the moment of the event; NULL when the event carries no content change (e.g. ingest-agent transitions). See [§23 Page Content Snapshots](#page-content-snapshots) for the full capture policy. |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n**Compound identifier expansion:** When a token contains underscores, the tokenizer emits both the compound form and each component as separate tokens (`capex_growth` → `[\"capex_growth\", \"capex\", \"growth\"]`). This expansion is applied at both index time and query time so that human-friendly queries match programmatic identifiers and vice versa. Leading/trailing underscores are stripped before splitting; no empty strings are emitted.\n\n**TF fallback for small corpora:** When all BM25 scores in a result set are zero or negative (IDF collapse on a corpus of ≤ 2 documents, or a term present in every page), the search system falls back to normalised term frequency (TF) scoring: `TF(doc, query) = Σcount(term, doc) / len(doc_tokens)`. Results are returned in the same format as BM25 results with a `tf_fallback: True` flag on each `SearchResult`. This field is recorded in the audit trail and consumed by gap detection (see below).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp, cascade_links_removed_from: [str]}` _(cascade field added v1.0.2)_ |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n| `GET` | `/graph` _(v1.0.0)_ | — | `{status, node_count, edge_count, cluster_count, nodes: [...], edges: [...]}` or `{status: \"computing\"}` on first call |\n| `GET` | `/pages/{slug}/history` | `?index=N&include_content=true` | List content snapshots; ?index=N&include_content=true for single |\n| `POST` | `/pages/{slug}/rollback` | `{index: int, reason: str}` | Restore page body to snapshot N |\n| `POST` | `/pages/{slug}/snapshot` | `{content: str, reason?: str}` | Record a content snapshot only when content differs from the last stored snapshot (`recorded: true/false`). Called by the Obsidian plugin on vault modify events. |\n| `DELETE` | `/pages/{slug}/history` | — | Delete all lifecycle events (including snapshots) for a slug; intended for test teardown. |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n**External file paths** (outside the wiki root) are supported when the request comes from\n`127.0.0.1` or `::1` and the payload includes `\"allow_external_paths\": true`. The CLI sets\nthis flag automatically for all local file sources. Remote clients cannot set this flag\n(the server ignores it for non-localhost connections) to prevent arbitrary file reads\nin server-exposed deployments.\n\nThis is required for ingesting files that live outside the wiki directory by design —\nfor example, Claude Code session transcripts at `~/.claude/projects//.jsonl`.\n\n**Client behaviour summary:**\n\n| Client | Sends `allow_external_paths` | Can ingest outside wiki root? |\n|--------|------------------------------|-------------------------------|\n| CLI (`synthadoc ingest `) | `true` (automatic, local file paths only) | Yes — server is localhost |\n| Obsidian plugin | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Web UI | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Direct HTTP API (localhost) | Set manually in request body | Yes — if request from 127.0.0.1/::1 |\n| Direct HTTP API (remote) | Ignored by server | No — server silently treats as `false` |\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 1.1.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n**Reading View default:** `synthadoc plugin install` and `synthadoc plugin upgrade` write `\"defaultViewMode\": \"preview\"` to `.obsidian/app.json` after copying the plugin files. This causes Obsidian to open new notes in Reading View by default, which is required for `^[filename:L-L]` citation chips to render. Only `defaultViewMode` is written; all other keys in `app.json` are preserved. If `app.json` is malformed JSON the write is skipped and a warning is logged — the install still succeeds.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Three-tab modal. **Current States** — sortable, filterable, paginated table (dynamic row count based on modal height) of all wiki pages with their current lifecycle state and last transition timestamp; click column headers to sort; each row shows valid transition buttons with a reason dialog before committing; draft/stale badge links on the lint modal and jobs panel open the table pre-filtered to that state. **Audit Log** — full history of every state transition across all pages, searchable by slug, filterable by target state, sortable by any column, with pagination and a purge footer (keep latest N per slug, or purge events before a date). **Content Snapshots** _(v1.2.0)_ — flat list of all snapshots across all pages, grouped by slug and sorted by index when unfiltered; filter-by-slug input with a × clear button; each row shows slug, snapshot index, state transition, triggered-by, timestamp, and size; click a row to open the corresponding wiki page; **View** button shows the snapshot diff against the current file body (LCS diff, YAML frontmatter stripped); **Rollback** button restores the page body to that snapshot (saves current body first, making rollback undoable). |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports two things: dangling slugs (pages listed in ROUTING.md that no longer exist in the wiki) and unassigned slugs (pages that exist in index.md but are not listed in any ROUTING.md branch). Enabled only when ROUTING.md exists. **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n| `Graph: show knowledge graph` _(v1.1.0)_ | Open the knowledge graph panel — draggable modal with Canvas force-directed graph, type filter dropdown, cluster legend, hover tooltip showing title/slug/type/state/cluster/connections, and click-to-open page in current pane. System pages (index, overview, dashboard, purpose, log) are excluded. Capped at 300 most-connected nodes for large wikis with an explanatory banner. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Background vault monitoring _(v1.2.0)_\n\nIn addition to the command palette commands, the plugin registers a passive vault event\nlistener for automatic content snapshot capture:\n\n- **`vault.on(\"modify\")`** — fires whenever Obsidian saves a file. The handler filters to\n `.md` files at the vault root only (wiki pages, not subdirectories or scaffold files).\n- **2-second per-slug debounce** — rapid keystrokes during editing collapse into a single\n request. The timer resets on each save event and only fires after 2 seconds of quiet.\n- **Deduplication on the server** — `POST /pages/{slug}/snapshot` calls\n `AuditLog.snapshot_if_changed()`, which compares the incoming content to the last\n stored snapshot. No new row is written when content is identical.\n- **Silent failure** — if the server is offline or the slug is not in the wiki, the error\n is swallowed. The plugin never shows an error notification for background snapshot calls.\n\nThis gives rollback coverage for edits that do not trigger a lifecycle transition — for\nexample, manually fixing an orphan page, editing body copy, or correcting a citation —\nwithout requiring any explicit user action.\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [-w wiki] — generate ROUTING.md from index.md branch structure\n│ ├── validate [-w wiki] — report dangling slugs and cross-branch duplicates\n│ └── clean [-w wiki] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [-w wiki]\n├── candidates\n│ ├── list [-w wiki]\n│ ├── promote |--all [-w wiki]\n│ └── discard |--all [-w wiki]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ ├── log [slug] [-w wiki] [--state ]\n│ ├── history [-w wiki] [--index N] [--show-content] list content snapshots (newest first)\n│ └── rollback [-w wiki] [--index N] [--reason \"\"] restore page body to a snapshot\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** subscription, or want to use the free **Opencode Zen** tier, you can use either as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode Zen (free, no API key — requires connecting first: run `opencode` → `/connect` → select Zen):\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"opencode/big-pickle\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 30\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n# Citation pass (Pass 4) tuning — these two settings work together:\n# citation_source_lines — how many lines of the source the LLM sees when placing ^[...] markers.\n# Increase if lint reports out_of_range on long sources (transcripts, PDFs).\n# citation_max_tokens — output token budget for the annotated section returned by the LLM.\n# Increase if you raise citation_source_lines and have long wiki sections.\n# citation_source_lines = 400\n# citation_max_tokens = 8192\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax`, `qwen` (DashScope), and `deepseek` providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `30` | Exponential backoff base; delays are `min(base × 2^(attempt−1), 300)` seconds |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Emit `logger.warning` in server log when per-job ingest cost exceeds this threshold; job continues normally |\n| `cost.hard_gate_usd` | float | `2.00` | Permanently fail the ingest job (DEAD) and record a `cost_gate_exceeded` audit event when per-job cost exceeds this threshold |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × (backup_count + 1)` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n| `ingest.max_source_chars` | int | `32000` | Character limit applied to each source before the LLM call. Sources exceeding this limit are truncated; the page's `sources:` frontmatter entry gets `truncated: true` and lint emits a warning. Override per-run with `--max-source-chars N`. _(v1.0.0)_ |\n| `ingest.citation_source_lines` | int | `400` | Number of source lines the LLM sees during Pass 4 (citation annotation). Increase if lint reports `out_of_range` on long sources such as transcripts or PDFs. Raise `citation_max_tokens` in proportion when increasing this value. |\n| `ingest.citation_max_tokens` | int | `8192` | Output token budget for the annotated section returned by Pass 4. Increase when `citation_source_lines` is raised or wiki sections are long, to avoid truncated annotations. |\n| `query.context_wiki_pct` | float | `0.60` | Fraction of the model context window reserved for wiki source pages. _(v1.0.0)_ |\n| `query.context_history_pct` | float | `0.20` | Fraction reserved for conversation history. _(v1.0.0)_ |\n| `query.context_system_pct` | float | `0.15` | Fraction reserved for system prompt and instructions. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_index_pct` | float | `0.05` | Fraction reserved for the wiki index preamble. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_window` | int | _(auto)_ | Override the built-in context window lookup for the configured model. Use when running a local or custom model whose window size is not in the built-in table. _(v1.0.0)_ |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement is wired into the ingest pipeline: `Orchestrator._guard_ingest_cost()` is called after each ingest LLM call with the returned cost, before the job is marked complete.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | `logger.warning` in server log; job continues normally |\n| `hard_gate_usd` | $2.00 | Server/ingest path: records `cost_gate_exceeded` audit event + permanently fails job (DEAD). Interactive CLI: prompts `Proceed? [y/N]` |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `min(backoff_base_seconds × 2^(attempt − 1), 300)` \nWith the default base of 30 s: attempt 1 waits 30 s, attempt 2 waits 60 s, attempt 3 waits 120 s (all capped at 300 s). The `retry_after` timestamp is stored in the jobs table; `dequeue()` skips any job whose `retry_after` is still in the future. Applied only to transient errors (network timeouts, connection failures, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` by default. The bind address is configurable via `server.host` in `config.toml` (e.g. `host = \"0.0.0.0\"` to expose on all interfaces for LAN access). No built-in authentication — restrict via firewall when exposing beyond loopback. Remote access without a reverse proxy is not recommended on shared or networked machines.\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs and unassigned slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `-w ` / `--wiki ` to target a specific wiki.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; when all local source files are missing from disk the page is transitioned to `archived` (if only some sources are missing it is transitioned to `stale` instead). Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: lint evaluates all local (non-URL) source paths listed in the page's frontmatter. If **all** of them are missing from disk, the page is transitioned to `archived`. If only **some** are missing — meaning valid content from at least one source still exists — the page is transitioned to `stale` instead, preserving it for re-ingest rather than discarding it. _(v1.0.2: previously any single missing source triggered an immediate archive.)_\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Cascade link cleanup\n\nWhenever a page transitions to `archived` — whether triggered manually (`synthadoc lifecycle archive`), via the Obsidian Lifecycle modal, the MCP `synthadoc_lifecycle` tool, or automatically by lint detecting a missing source — Synthadoc immediately scans all other active pages and removes every `[[archived-slug]]` wikilink pointing to the now-archived page.\n\n- **Inline links** — `[[archived-slug]]` becomes plain text (the display text if an alias was used, otherwise the slug itself).\n- **List-item links** — a list item whose only content is `[[archived-slug]]` is dropped entirely.\n- **Archived pages are skipped** — pages already archived are not rewritten.\n- **System pages are skipped** — index, overview, dashboard, log, and purpose pages are excluded.\n\nThe cleanup runs synchronously before the response is returned, so the wiki is consistent immediately. The CLI and MCP responses include a `cascade_links_removed_from` field listing the affected slugs. During `lint run`, cascade is batched once after all auto-archive transitions complete, and the count is added to `dangling_links_removed` in the lint report.\n\nLint's existing periodic dangling-link cleanup (`lint run --scope orphans`) remains in place as a safety net for any dead links that predate this feature or arrive through other paths.\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc lifecycle history -w [--index ] [--show-content] [--json]\n List content snapshots captured at lifecycle transitions (newest first).\n --index inspects a single snapshot; --show-content prints the full body.\n\nsynthadoc lifecycle rollback -w --index --reason \"...\"\n Restore the page body from snapshot . The current body is saved as a new\n snapshot first, so every rollback is undoable.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n### Page Content Snapshots\n\nA content snapshot is a copy of the page body (`WikiPage.content`) stored in the\n`content_snapshot` column of a `lifecycle_events` row. Three distinct triggers produce\nsnapshots; ingest-agent transitions never do (ingest does not change existing content):\n\n| Trigger | When it fires | Dedup? |\n|---------|--------------|--------|\n| **Manual lifecycle transition** | User calls `lifecycle activate`, `lifecycle archive`, `lifecycle restore` via CLI, the Obsidian Lifecycle modal, the REST API, or the MCP tool | No — always records |\n| **Lint-driven state change** | `LintAgent._transition()` fires: draft → active (promotion), active → stale (hash mismatch), active → archived (source gone), contradicted → active (auto-resolve) | No — always records; for auto-resolve, captures the post-resolution body that was just written |\n| **Manual file edit in Obsidian** | The Obsidian plugin's `vault.on(\"modify\")` handler fires after a 2-second debounce on any `.md` file at the vault root | Yes — `AuditLog.snapshot_if_changed()` compares incoming content to the last stored snapshot and only writes a new row when content differs |\n\nFor the manual-edit trigger the `from_state` and `to_state` columns are both set to the\ncurrent page state (no lifecycle transition occurs — this is a pure content checkpoint).\n\n**Deduplication (manual edits only):** `snapshot_if_changed()` reads the last\n`content_snapshot` for the slug from the DB and compares it to the incoming content with\na string equality check. At typical wiki page sizes (< 100 KB) and the 2-second debounce\nrate, this is negligible overhead. A future optimisation may store a `content_hash`\ncolumn to avoid reading the full text from the DB; deferred until page sizes make it\nmeasurable.\n\nSnapshots are indexed 1-based, newest first. Index 1 always refers to the most recent\nsnapshot. The rollback operation:\n\n1. Records the current body as a new snapshot (making rollback undoable).\n2. Writes the target snapshot body to the `.md` file on disk.\n3. Does **not** change `page_states` or call `set_page_state`.\n\nStorage: ~3 KB per snapshot. Existing `purge_lifecycle_events()` removes rows\n(and their snapshots) on `--before` or `--keep-latest` purge.\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [...], \"input_tokens\": 1240, \"output_tokens\": 380, \"tokens_used\": 1620}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […], \"input_tokens\": N, \"output_tokens\": N, \"tokens_used\": N}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Streaming Token Audit\n\nStreaming LLM responses do not return token counts in the same way as blocking calls. To track real usage in the audit trail:\n\n**Provider-side collection** — each `LLMProvider` subclass exposes two instance attributes: `last_stream_input_tokens` and `last_stream_output_tokens`. These default to `0` and are populated from the API-specific usage field when the stream ends:\n\n| Provider | Mechanism | Verified |\n|---|---|---|\n| OpenAI | `stream_options={\"include_usage\": True}` passed to `chat.completions.create()`; final chunk carries `chunk.usage.prompt_tokens` / `completion_tokens` with empty `choices` | ✅ |\n| Anthropic | `message_start` event → `event.message.usage.input_tokens`; `message_delta` event → `event.usage.output_tokens` | ✅ |\n| Ollama | Final chunk with `done=True` → `prompt_eval_count` / `eval_count` | ✅ |\n| DeepSeek | Same `OpenAIProvider` path as OpenAI; DeepSeek's OpenAI-compatible API supports `stream_options` | ✅ |\n| MiniMax (reasoning: M2.5+) | Detects `` in stream → falls back to blocking `complete()` → captures exact counts from `resp.usage` | ✅ |\n| MiniMax (non-reasoning, e.g. M3 thinking=disabled) | Same `OpenAIProvider` path; MiniMax API silently ignores `stream_options`. Falls back to character-based estimate (÷ 3.5 chars/token) from prompt + answer lengths. Accuracy ±20%. | ✅ estimated |\n| Gemini | Same `OpenAIProvider` path via `generativelanguage.googleapis.com/v1beta/openai/`; Google's compatibility layer honours `stream_options` — verified live (Gemini 2.5 Flash Lite, 50K tokens) | ✅ |\n| Groq | Same `OpenAIProvider` path; Groq's OpenAI-compatible API supports `stream_options` | ✅ |\n| Qwen (DashScope) | Same `OpenAIProvider` path via DashScope; `stream_options` honoured — verified live (qwen-plus, 28K tokens) | ✅ |\n| Qwen (Ollama) | Ollama path → `prompt_eval_count` / `eval_count` | ✅ |\n\n**Forwarding to `done` event** — after `complete_stream()` is exhausted, `QueryAgent.run_stream()` reads the provider attributes and includes them in the final `done` event payload: `input_tokens`, `output_tokens`, `tokens_used`.\n\n**Audit record** — `Orchestrator.query_stream()` consumes the `done` event and calls `estimate_cost(model, input_tokens, output_tokens, is_local)` from `synthadoc/providers/pricing.py`, then calls `AuditDB.record_query()` with the real token count and computed cost. The audit row at `GET /audit/queries` will therefore always show non-zero values for cloud providers.\n\nWhen a provider does not report streaming usage (e.g. a future provider not yet wired), the attributes stay at `0`; the audit row records `tokens=0, cost_usd=0.0` rather than crashing, matching the pre-fix behaviour.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 5) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 5 # turns to include in each request (default: 5; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp, cascade_links_removed_from: [str]}` or `{error, cascade_links_removed_from: []}` _(cascade field added v1.0.2)_ | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions for Codex CLI / OpenCode (if present)\n├── CLAUDE.md ← same content; loaded by Claude Code (if present)\n├── GEMINI.md ← same content; loaded by Gemini CLI (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── hooks/ ← user hook scripts (always included)\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ ├── extracted/ ← text sidecars + PDF pagemaps (always included)\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `logs/`.\n\n> **v0.9.3 fix:** `hooks/` and `.synthadoc/extracted/` were previously missing from backups. Omitting `extracted/` caused the Source Viewer and Provenance citation modal to show \"Could not read source file\" after restore.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## 29. Pre-LLM Source Sanitizer\n\nEvery source document passes through a sanitizer immediately after text extraction, before any LLM call. The sanitizer removes six categories of content that could manipulate the LLM or degrade compilation quality:\n\n| Category | Action | Warning logged? |\n|---|---|---|\n| Zero-width characters (U+200B, U+200C, U+200D, U+FEFF) | Removed silently | No |\n| Bidi override characters (U+202A–U+202E, U+2066–U+2069) | Removed | Yes |\n| HTML comments (``) | Removed silently | No |\n| Hidden CSS spans (`display:none`, `visibility:hidden`) | Removed silently | No |\n| Base64 blobs ≥ 200 consecutive characters | Replaced with `[base64 content removed]` | Yes |\n| Instruction-override phrases (8 patterns: \"ignore previous instructions\", \"disregard the above\", \"override your system prompt\", etc.) | Replaced with `[redacted]` | Always |\n\nWhen content is removed and a warning is appropriate, the server logs a `WARN` entry:\n```\n[WARN] sanitizer stripped content from 'papers/survey.pdf': bidi overrides, instruction-override phrase\n```\n\nThis is a pure pre-processing step with zero LLM cost. Truncation (if configured) is applied after sanitization, so the character budget is consumed by clean text only.\n\n---\n\n## 30. Per-Source Truncation Flag\n\nLarge source documents are truncated before they reach the LLM. By default the limit is **32,000 characters** (~8,000 tokens), raised from the prior hardcoded 8,000-character limit. When a source exceeds this limit, its compiled wiki page records a `truncated: true` flag in its `sources:` frontmatter entry.\n\n### Configuration\n\n```toml\n[ingest]\nmax_source_chars = 32000 # default; raise for large PDFs or books\n```\n\n### Per-ingest override\n\nThe limit can be raised for a single ingest run without modifying the config:\n\n```bash\nsynthadoc ingest papers/large-textbook.pdf --max-source-chars 128000\n```\n\nThe same `max_source_chars` field is accepted by the HTTP `POST /jobs/ingest` body and the MCP `synthadoc_ingest` tool.\n\n### Lint surfacing\n\n`synthadoc lint` emits a warning for any page that has a truncated source:\n\n```\n[WARN] quantum-computing.md: source 'papers/large-survey.pdf' was truncated at ingest\n (source exceeded max_source_chars=32000 — 87,412 chars in source).\n To re-ingest with a higher limit (this source only):\n synthadoc ingest papers/large-survey.pdf --max-source-chars 128000\n To raise the limit for all future ingests:\n set [ingest] max_source_chars = 128000 in your config\n```\n\n---\n\n## 31. Proportional Context Budget\n\nQuery context is allocated proportionally to the configured model's context window, replacing the prior hard cap. The default allocation:\n\n| Slice | Default | Purpose |\n|---|---|---|\n| `context_wiki_pct` | 60% | Wiki page content (ranked by BM25 + vector score) |\n| `context_history_pct` | 20% | Chat turn history (newest-first, oldest dropped when over budget) |\n| `context_system_pct` | 15% | System prompt and purpose document |\n| `context_index_pct` | 5% | `ROUTING.md` and search index |\n\nPercentages must sum to ≤ 100. Configure in `[query]`:\n\n```toml\n[query]\ncontext_window = 0 # 0 = auto-detect from known model map\ncontext_wiki_pct = 60\ncontext_history_pct = 20\ncontext_system_pct = 15\ncontext_index_pct = 5\n```\n\n> **Note:** `context_system_pct` and `context_index_pct` are parsed and validated but not yet enforced as hard caps in this release — the system prompt uses a conservative fixed limit, which is well within the configured 15% slice for all supported models. Full per-slice enforcement is planned for v1.1.\n\n### Model context window map\n\nSynthadoc ships a built-in prefix-matched table:\n\n| Model | Context window |\n|---|---|\n| `claude-opus-4*`, `claude-sonnet-4*`, `claude-haiku-4*` | 200,000 tokens |\n| `gpt-4o*`, `gpt-4-turbo*` | 128,000 tokens |\n| `gpt-4` (exact) | 8,192 tokens |\n| `gpt-3.5-turbo*` | 16,385 tokens |\n| Unknown / fallback | 128,000 tokens |\n\nSet `context_window = N` in config to override the map (useful for local Ollama models with non-standard context sizes).\n\nAt the defaults on a 200k-token model: wiki slice ≈ 480,000 chars — far more than any practical query needs. On constrained models, the budget scales down automatically so the same config works across model families.\n\n---\n\n## 32. Knowledge Graph\n\nSynthadoc computes a wikilink graph across all active and draft pages during every lint run. The graph is stored in two tables in `audit.db`:\n\n```sql\nCREATE TABLE graph_nodes (\n slug TEXT PRIMARY KEY,\n cluster_id INTEGER NOT NULL,\n updated_at TEXT NOT NULL\n);\n\nCREATE TABLE graph_edges (\n from_slug TEXT NOT NULL,\n to_slug TEXT NOT NULL,\n weight INTEGER NOT NULL DEFAULT 1,\n edge_type TEXT NOT NULL DEFAULT 'mixed',\n PRIMARY KEY (from_slug, to_slug)\n);\n```\n\nEdge weight is a composite signal from two sources:\n\n| Edge type | Source | Weight contribution |\n|-----------|--------|---------------------|\n| `wikilink` | `[[slug]]` occurrences in page body | +1 per occurrence |\n| `co_source` | pages compiled from the same source file (matched by hash) | +2 per shared source |\n\nMost edges are `mixed` (have both types). Pure `co_source` edges surface hidden relationships — pages compiled from the same source document are linked immediately after ingest, before any wikilinks are created. `edge_type` is stored on each edge so the web UI can render co-source edges with a different visual style.\n\nCluster IDs are assigned by Louvain community detection — pages that link densely to each other end up in the same cluster.\n\n### REST API\n\n**`GET /graph`** — returns the full graph with per-node enrichment:\n\n```json\n{\n \"status\": \"ready\",\n \"node_count\": 45,\n \"edge_count\": 123,\n \"cluster_count\": 4,\n \"nodes\": [\n {\"slug\": \"quantum-error-correction\", \"title\": \"Quantum Error Correction\",\n \"type\": \"concept\", \"state\": \"active\", \"cluster_id\": 2}\n ],\n \"edges\": [\n {\"from\": \"quantum-error-correction\", \"to\": \"shor-algorithm\", \"weight\": 3, \"edge_type\": \"mixed\"}\n ]\n}\n```\n\nWhen the graph tables are empty (first call after upgrade, or before the first lint run), the server returns `{\"status\": \"computing\"}` and starts a background build immediately. The web UI polls every 2 seconds and renders the graph when the status becomes `\"ready\"`.\n\n### Web UI\n\nThe **Graph** tab (alongside Chat in the top nav) renders a D3.js force-directed graph. Nodes are colored by cluster. Edge thickness is proportional to weight (1–4 px). Pure `co_source` edges (no wikilink) are rendered with a dashed stroke, making hidden source-based relationships visually distinct from explicit cross-links. Clicking a node opens a sidebar showing the page title, type and lifecycle state badges, cluster assignment, and an **\"Ask about this →\"** button that switches to the Chat tab with a pre-filled query.\n\nControls: zoom in/out (scroll or pinch), drag nodes, filter by page type.\n\n### System page exclusion _(v1.1.0)_\n\nThe five structural system pages — `overview`, `index`, `dashboard`, `purpose`, and `log` — are excluded from both `graph_nodes` and `graph_edges`. These pages link to nearly every other page and would act as false hubs, inflating edge counts and distorting Louvain cluster detection. System page exclusion is applied inside `_build_graph()` before any edge computation.\n\n### Obsidian knowledge graph panel _(v1.1.0)_\n\n→ Full description: [§33 Multi-Platform Agent Skill Files](#33-multi-platform-agent-skill-files) covers scaffold; the graph panel itself is documented in [§8 Obsidian Plugin](#8-obsidian-plugin) command table.\n\nThe Obsidian plugin includes a **draggable modal** knowledge graph panel (`Graph: show knowledge graph` command). It fetches from the same `GET /graph` endpoint as the web UI and renders a force-directed Canvas 2D simulation using Verlet integration (no external library). System pages are already excluded from the server response.\n\n| Feature | Obsidian panel | Web UI Graph tab |\n|---------|----------------|-----------------|\n| Engine | Canvas 2D + Verlet | D3.js SVG |\n| Node click | Opens the wiki page in the current pane | Opens the node sidebar with \"Ask about this →\" |\n| **Best for** | **Navigating to and editing pages** | **Asking questions about a page** |\n| Draggable panel | Yes — drag header to reposition | Fixed tab layout |\n| Node cap | 300 most-connected (see note) | No cap |\n| Cluster legend | Yes | Yes |\n| Type filter | Yes | Yes |\n\n**Rule of thumb:** Use the **web UI** when you want to ask questions about a node. Use the **Obsidian panel** when you want to navigate to and edit a page.\n\n**Node cap rationale:** The Obsidian panel uses a hand-written Verlet integrator with O(n²) pairwise charge repulsion — every node repels every other node on each animation tick. At 300 nodes that is ~90,000 operations per frame, which runs smoothly at 60 fps inside Obsidian's Electron renderer. At 500 nodes it is ~250,000 operations per frame and starts to lag noticeably. The web UI has no cap because D3's `forceManyBody` uses a Barnes-Hut tree approximation (O(n log n)), which scales to thousands of nodes without frame-rate issues. When the cap is hit, nodes are ranked by degree and the 300 most-connected are shown; a banner reports the total page count and the number of hidden nodes, and suggests using the type filter to bring the set below 300.\n\n→ User walkthrough: [Quick-Start Guide §24 — Knowledge Graph](docs/user-quick-start-guide.md#knowledge-graph)\n\n---\n\n## 33. Multi-Platform Agent Skill Files\n\n\n\n`synthadoc init` and `synthadoc scaffold` write three agent skill files at the wiki root:\n\n| File | Format | Read by |\n|------|--------|---------|\n| `AGENTS.md` | OpenAI Agents SDK / Codex / OpenCode | Codex CLI, OpenCode, generic agents |\n| `CLAUDE.md` | Claude Code / Anthropic | Claude Code (loaded automatically when the wiki folder is opened) |\n| `GEMINI.md` | Gemini CLI | Gemini CLI |\n\nAll three files carry **identical content**: a title block, the LLM-generated domain guidelines for the wiki, a quick-reference CLI command table, and an MCP tool table. The only difference is the heading comment. This means any AI coding tool that opens the wiki root gets full Synthadoc guidance without manual setup.\n\n### Content generated by `scaffold`\n\nEach skill file is generated from the same LLM call that produces `purpose.md` and `index.md`. The LLM receives the current wiki page list and domain description, then generates:\n\n1. **Domain label** — a precise human-readable name inferred from wiki content (e.g. `History of Computing`, not `My Wiki`)\n2. **Domain guidelines** — 3 domain-specific ingest and query guidelines as plain sentences\n3. **Quick Reference table** — pre-filled with the current wiki name and port\n\nIf the wiki was created before v1.1.0, running `synthadoc scaffold` once regenerates all three files and adds `CLAUDE.md` and `GEMINI.md` to any wiki that only had `AGENTS.md`.\n\n### Template source\n\nThe instruction body is defined in `synthadoc/cli/_init.py` as `_AGENT_INSTRUCTION_BODY` and shared by all three templates, keeping the files permanently in sync. The scaffold agent substitutes `{domain}`, `{guidelines}`, and `{port}` at write time.\n\n---\n\n## 34. Synthadoc Scaffold Marker\n\n\n\nThe scaffold marker `` (written as an HTML comment) separates **user-authored content** (preserved) from **scaffold-managed content** (rewritten on each `synthadoc scaffold` run).\n\n### Single-marker mode (index.md)\n\nThe marker appears once, below the H1 heading in `index.md`. Everything above the marker is the user zone; everything below is replaced with new LLM-generated categories.\n\n```markdown\nMy custom introduction — maintained by hand.\n\n\n\n## Pioneers and Visionaries\n- [[alan-turing]]\n...\n```\n\nIf the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### Per-section multi-marker mode (purpose.md)\n\n`purpose.md` uses a **per-section** approach: the marker appears inside each `## Section` heading. Within a section, content the user writes **above** the marker is preserved; content below is replaced with fresh LLM output. Sections without a marker in the existing file are replaced entirely (and a marker is added). Sections in the existing file that the scaffold template does not include are kept unchanged.\n\n```markdown\n## Overview\n\nMy custom overview note.\n\n\n\nLLM-generated overview content goes here.\n\n## What Belongs\n\n\n\nLLM-generated scope definition.\n```\n\nThis design lets users annotate any section of `purpose.md` without losing their notes on the next scaffold run.\n\n### Marker preservation guarantee\n\n- `preserve_user_zone()` in `scaffold_agent.py` handles both modes.\n- If a section has no marker in the existing file, `preserve_user_zone()` treats the entire section body as user content and preserves it, replacing only what falls below any marker it finds.\n- The scaffold NEVER touches `config.toml`, `dashboard.md`, or any wiki page (`wiki/*.md`).\n\n→ User walkthrough: [Quick-Start Guide §14 — Enrich the wiki with scaffold](docs/user-quick-start-guide.md#scaffold)\n\n---\n\n## Customization\n\nSynthadoc exposes four extension points — all hot-loaded, no server restart required.\n\n### Custom skills (new file formats)\n\nSubclass `BaseSkill` (Apache-2.0 — no AGPL obligation on your skill code), drop the folder in `/skills/` or `~/.synthadoc/skills/`, and Synthadoc picks it up on the next ingest. Skills match by file extension or intent prefix and support any Unicode text, including CJK prefixes.\n\n→ Full interface, manifest format, and examples: [§5 Skills System](#5-skills-system) and [§17 Plugin Development Guide](#17-plugin-development-guide)\n\n### Custom LLM providers\n\nSubclass `LLMProvider` from `synthadoc/providers/base.py` (Apache-2.0) and place the file in `~/.synthadoc/providers/` or the wiki `providers/` directory. Switch active provider with one config line.\n\n→ Full interface and wiring instructions: [§10 Configuration — Provider switching](#10-configuration)\n\n### Hooks\n\nShell scripts (any language) that fire on `on_ingest_complete` and `on_lint_complete`. Receive a JSON context on stdin. Set `blocking = true` to gate the operation on the hook's exit code — useful for CI pipelines and quality gates.\n\n→ Full event schema, context JSON, and blocking behaviour: [§11 Hook System](#11-hook-system)\n\n### Cache control\n\nThree cache layers (embedding, LLM response, provider prompt cache) invalidate automatically on source-file change (SHA-256). Force a fresh call with `--force`, or wipe all cached responses with `synthadoc cache clear -w `.\n\n→ Layer-by-layer breakdown and invalidation rules: [§12 Cache System](#12-cache-system)\n\n### Multi-Platform Agent Skill Files\n\nEvery Synthadoc wiki ships three companion files that give AI coding agents a complete, self-contained operating guide for the wiki:\n\n| File | Platform |\n|---|---|\n| `AGENTS.md` | Codex CLI, OpenCode, and any agent that reads the AGENTS.md convention |\n| `CLAUDE.md` | Claude Code (this session's tool) — highest-priority instruction source in Claude's hierarchy |\n| `GEMINI.md` | Gemini CLI — relevant because Synthadoc's default LLM provider is `gemini-2.5-flash-lite` |\n\nAll three files share identical body content generated from the same template; they differ only in their H1 heading so each agent recognises its own file first. The body covers:\n\n- **Domain guidelines** — LLM-generated bullet list of domain-specific ingest and query rules (produced by `scaffold`)\n- **Quick reference table** — every common CLI command with the correct `synthadoc` syntax\n- **Server startup** — how to start the server and verify it is running\n- **Ingest examples** — local file, URL, YouTube, agent session `.jsonl`, force reingest, dry-run\n- **Query** — basic and streaming query syntax; citation marker format\n- **Lint** — when to run and what each check does\n- **Lifecycle** — transition commands with state names\n- **Page schema** — frontmatter structure with all fields\n- **MCP tools table** — all 12 tools with purpose descriptions\n\n**How the files are created and updated:**\n\n1. `synthadoc init` — writes all three files with default domain guidelines and the wiki's configured port.\n2. `synthadoc scaffold` — regenerates all three with LLM-produced domain guidelines derived from the current wiki state. The `scaffold` command prints a confirmation line for each file.\n\n**Use cases:**\n\n- Open Claude Code in a Synthadoc wiki directory and it immediately knows how to ingest, query, lint, and manage lifecycle — no additional setup. Claude Code can also start the server in background mode (`synthadoc serve -w -b`), stop it cleanly, and orchestrate multi-step workflows, effectively acting as a control plane for the wiki.\n- Gemini CLI reads `GEMINI.md` and can run `synthadoc` commands with the correct flags without asking the user for help.\n- CI pipelines with Codex CLI use `AGENTS.md` to drive automated lint and lifecycle transitions on PR merge.\n- Agent session files produced by Claude Code (`.jsonl` under `~/.claude/projects/`) can themselves be ingested into the wiki, capturing the reasoning behind design decisions alongside the source documents.\n\n→ See [README.md — Interfaces & Integration](../README.md#interfaces--integration) for the feature comparison row and quick-start instructions.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v1.1.2\n\n- **Exponential backoff on transient URL fetch failures** — jobs failing with transient errors (connection timeouts, 5xx responses, dropped TCP connections including `RemoteProtocolError`) now re-queue with exponential backoff: `min(base × 2^(attempt−1), 300)` seconds (default base 30 s). Previously all failures retried immediately, exhausting the retry budget within seconds. `dequeue()` skips any job whose `retry_after` timestamp is still in the future; other jobs run normally during the wait window. See [§14 Job Queue](#14-job-queue).\n- **Accurate retry outcome in job warning logs** — the transient-failure warning now reads the current attempt count from the database before logging, so it correctly reports `will retry` vs. `final attempt` (was always `will retry` regardless of retry state).\n- **Scheduler subprocess timeout** — scheduled jobs (`lint`, `ingest --batch`, etc.) are now guarded by `asyncio.wait_for(proc.communicate(), timeout=job_timeout_seconds)`; a hung subprocess is killed after the timeout and the run is recorded as `failed`. Previously a stalled subprocess could block the scheduler task indefinitely. `job_timeout_seconds` defaults to 600 s and is now shown (uncommented) in fresh-init `config.toml`.\n- **Streaming query token and cost accuracy** — the streaming query audit record was written with `tokens=0, cost_usd=0.0`; it now records the actual LLM token usage and cost from the streamed response.\n- **Cost hard gate timing** — the cost hard gate now fires before LLM tokens are spent and pages are written; previously the gate ran after the LLM call completed, allowing partial page writes before rejection.\n- **Session state memory growth bounded** — `_session_state` in the HTTP server is pruned at 100 concurrent sessions (LRU eviction) to prevent unbounded memory growth on long-running servers.\n- **Streamed `` block cleanup** — a leaked fallback asyncio task is now cancelled when a streaming response ends inside an unclosed `` block (e.g. reasoning-model responses cut off mid-think).\n- **Config validation for staging fields** — `staging_policy` values other than `off`, `all`, and `threshold`, and non-numeric `staging_confidence_min`, were silently accepted and caused runtime errors; they now raise a `ConfigError` at server startup.\n- **Job status guard on retry** — retrying a job that is already `in_progress`, `done`, or `skipped` now returns an error instead of silently resetting it to `pending`.\n- **Jobs endpoint performance** — `GET /jobs/{id}` and `DELETE /jobs/{id}` now use a direct primary-key lookup instead of a full table scan; eliminates O(N) reads on large job queues.\n- **Vector migration GC safety** — the background vector migration `asyncio.Task` is now held in a strong reference to prevent premature garbage collection mid-run.\n- **Quota error stack trace suppression** — Qwen 403 errors and Claude Code session-limit exhaustion now emit a concise warning naming the actionable config key, instead of a full stack trace.\n- **Scaffold agent JSON robustness** — `MiniMax-M3` and other reasoning models can truncate mid-JSON; the scaffold agent now handles truncated JSON output gracefully rather than failing the job.\n- **Staging/Candidates section in agent skill files** — `AGENTS.md`, `CLAUDE.md`, and `GEMINI.md` generated by `synthadoc scaffold` now include a `## Staging and Candidates` section.\n- **Citation failure pagination fix** — `GET /audit/citation-failures` was applying the `LIMIT` in SQL before filtering by `page_slug`, returning incorrect results for paginated per-page queries; filtering now runs before the limit.\n\n### v1.1.0\n\n- **Weighted knowledge graph edges** — graph edges now carry two signals: wikilink occurrences (+1 each) and co-source connections — pages compiled from the same source file (matched by SHA-256 hash) — (+2 per shared source). `edge_type` field added to `graph_edges` table (`wikilink`, `co_source`, or `mixed`). `GET /graph` exposes `edge_type` per edge. Web UI renders edge thickness proportional to weight (1–4 px) and dashed lines for pure co-source edges. Co-source edges appear immediately after ingest — before any wikilinks — surfacing hidden relationships in the graph. Schema migrated to version 3.\n- **Multi-platform agent skill files** — `synthadoc init` and `synthadoc scaffold` write `CLAUDE.md`, `AGENTS.md`, and `GEMINI.md` at wiki root; all three carry LLM-generated domain guidelines and the full CLI quick-reference so Claude Code, Codex CLI, OpenCode, Pi, and Gemini CLI users get first-class guidance without manual setup. Run `synthadoc scaffold` once on any pre-v1.1.0 wiki to generate the new files. See [§33](#33-multi-platform-agent-skill-files).\n- **Session history ingestion skill** — `.jsonl` session files from Claude Code and Codex CLI are ingested as wiki pages; human turns and substantive assistant responses extracted; tool calls, thinking blocks, and sub-agent scaffolding skipped; format auto-detected; `suggested_slug` derived from session date and first user message. Converts ephemeral problem-solving knowledge into searchable wiki entries. See [§5 Skills System — Session Skill](#session-skill--ai-session-history-ingestion).\n- **Obsidian knowledge graph panel** — new `Graph: show knowledge graph` command in the Obsidian plugin opens a draggable modal with a Canvas 2D force-directed graph. Features: Verlet integration, type filter dropdown, cluster legend, hover tooltip (title, slug, type, state, cluster, connections), click-to-open page in current pane, node drag to repin. Closes only via the × button — Escape and outside-click do not close the panel, matching the behaviour of other Synthadoc modals. **Large-wiki cap:** when the node set exceeds 300 (after any type filter is applied), nodes are ranked by degree (total edge count) and the 300 most-connected are displayed; edges to capped-out nodes are removed; a banner explains the cap and suggests using the type filter to narrow the view below 300. See [§8 Obsidian Plugin](#8-obsidian-plugin) and [§32 Knowledge Graph — Obsidian panel](#obsidian-knowledge-graph-panel-v110).\n- **Per-section scaffold markers in `purpose.md`** — `purpose.md` now uses per-section `` markers. Users can write content above the marker in any `## Section` block; that content is preserved on every re-scaffold. A single-marker mode for `index.md` continues to work as before. See [§34 Synthadoc Scaffold Marker](#34-synthadoc-scaffold-marker).\n- **System pages excluded from knowledge graph** — `overview`, `index`, `dashboard`, `purpose`, and `log` are filtered out of `_build_graph()` before any edge computation so they cannot act as false hubs in Louvain cluster detection. Both the Obsidian graph panel and the web UI Graph tab reflect this exclusion.\n- **Web UI graph cluster legend** — the web UI Graph tab now shows a cluster legend below the canvas (matching the Obsidian plugin panel), with a color swatch and label for each Louvain cluster present in the filtered node set.\n- **`_merge_section_into_page` deduplication** — force re-ingest of the same source file (e.g. an updated `.jsonl` session) now replaces the page body rather than appending a duplicate section. `_files_match()` normalizes absolute vs. relative paths so the same file is recognized regardless of how it was passed to the ingest command.\n- **`purpose.md` domain label inference** — `scaffold` now infers a precise `domain_label` from current wiki content (e.g. `History of Computing`) instead of using a generic placeholder; the label is used in all three agent skill files and in `purpose.md`'s Overview section.\n- **CI: Windows runner fix** — `scripts/sync_plugin.py` and `scripts/sync_web_ui.py` now use `shutil.copy2` instead of `cp -f`, resolving CI failures on Windows runners for the sync check.\n\n### v1.0.2\n\n- **Cascade link cleanup on archive** — archiving any page immediately removes every `[[slug]]` wikilink pointing to it from all other active pages, without waiting for the next `lint run`. Inline links become plain text; list-item-only links are dropped entirely. The HTTP `POST /lifecycle/transition` and MCP `synthadoc_lifecycle` responses now include a `cascade_links_removed_from` field listing the rewritten page slugs. During `lint run`, cascade runs once after all auto-archive transitions complete and the count is included in the lint report's `dangling_links_removed` field. System pages (index, dashboard, purpose, log, overview) and already-archived pages are skipped. The existing dangling-link lint pass remains as a safety net for dead links that predate this release.\n- **Multi-source partial-archive fix** — when a page has multiple local source files and only some are missing from disk, lint now transitions the page to `stale` (not `archived`), preserving it for re-ingest. The page is only archived when every local source file is gone. URL sources are not affected by this check.\n\n### v1.0.1\n\n- **Scaffold JSON extraction hardened** — replaced the greedy regex used to extract the LLM's JSON response with a brace-balanced parser; eliminates false-positive matches on `[[wikilink]]` syntax that previously caused `json.loads()` failures and silent scaffold corruption\n- **Generated file validation** — scaffold now validates every LLM-generated file (index, purpose, AGENTS.md) for minimum length, required frontmatter, and `[[wikilink]]` presence before writing to disk; validation failure marks the job as failed with a structured error message rather than silently writing a malformed page\n- **ROUTING.md regeneration validation** — `routing init` and scheduled routing regeneration now validate the regenerated file for minimum line count and anchor presence before applying it; prevents a corrupt or truncated ROUTING.md from entering the wiki\n- **Scaffold failure reporting** — scaffold skip and failure reasons are now surfaced in the terminal output and job error field; previously silent failures (e.g. LLM timeout, JSON parse error) are now visible without tailing the JSON log\n- **Install-time scaffold removed** — `synthadoc install` no longer runs the LLM scaffold call during installation; scaffold is a separate step (`synthadoc scaffold`) that the user runs after setting their API key, avoiding confusing API-key-not-set errors at install time\n- **AquaFlow LLM evaluation** — 15-question M&A due-diligence benchmark across five models (MiniMax-Think M3, Claude Opus 4.8, Claude Sonnet 4.6, DeepSeek-R1, Qwen Plus); full report at `docs/example/aquaflow/evaluation/report/llm-query-benchmark.md`\n\n### v1.0.0\n\n- **Pre-LLM source sanitizer** — every source document passes through a sanitizer before any LLM call; removes six categories of potentially harmful content (zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars, instruction-override phrases); a `WARN` log entry is emitted when bidi overrides or instruction-override phrases are stripped; zero LLM cost; applied before truncation so the character budget is consumed by clean text only\n- **Per-source truncation flag** — default character limit raised from 8,000 to 32,000 characters per source (~8,000 tokens); truncated sources are flagged with `truncated: true` in the page's `sources:` frontmatter; `--max-source-chars N` CLI flag and matching `POST /jobs/ingest` body field and MCP `synthadoc_ingest` parameter override the limit per run; `synthadoc lint` emits a warning for any page with a truncated source and suggests the override command; configurable via `[ingest] max_source_chars` in `config.toml`\n- **Proportional context budget** — query context is allocated proportionally to the configured model's context window; four configurable slices: `context_wiki_pct` (60%), `context_history_pct` (20%), `context_system_pct` (15%), `context_index_pct` (5%); built-in prefix-matched model context window table covers Claude 4, GPT-4o/turbo/3.5, and GPT-4; unknown models fall back to 128,000 tokens; `context_window = N` in `[query]` overrides the table for local or custom models\n- **Knowledge graph** — wikilink graph computed across all active and draft pages during every lint run; stored in `graph_nodes` and `graph_edges` tables in `audit.db`; edge weight = number of `[[wikilink]]` references between two pages; cluster IDs assigned by Louvain community detection; `GET /graph` REST endpoint returns nodes with per-node enrichment (title, type, state, cluster) and edges; first call after upgrade triggers a background build and returns `{\"status\":\"computing\"}`; web UI **Graph** tab (D3.js force-directed graph) — nodes colored by cluster, click to see page details and an \"Ask about this →\" button that opens a pre-filled chat query\n- **Citation accuracy & rendering improvements** — six correctness fixes to the ingest citation pipeline: (1) full source text passed to the decision LLM so citations can reference any line in the document; (2) `_annotate_citations` bugs A–F resolved (false-positive matches, off-by-one line ranges, duplicate markers, escaped bracket handling, multi-paragraph span errors, and source filename normalisation); (3) `_has_citations` check added — lint warns and suggests a model upgrade when a page has zero citation markers after synthesis; (4) `Key Data` section extraction hardened so numerical facts, rates, and formulas are preserved verbatim across the annotation pass; (5) `obsidianCitationsToGfm` rendering fixed in the web UI — citation superscripts now display correctly in the chat panel alongside streamed answers; (6) reading view set as the default Obsidian display mode for wiki pages\n\n### v0.9.3\n\n- **Backup now includes `hooks/` and `.synthadoc/extracted/`** — both directories were silently omitted from backups. `hooks/` contains user-customised hook scripts; `.synthadoc/extracted/` contains per-source text sidecars and PDF pagemaps written by the ingest agent. Without these, the Source Viewer and Provenance citation modal showed \"Could not read source file\" after restore.\n- **PyPI publish workflow** — `publish.yml` triggers on GitHub Release, builds the wheel with hatchling, and uploads to PyPI via OIDC trusted publishing (no API token stored). `pip install synthadoc` now works.\n- **Web UI bundled in pip package** — `synthadoc/data/web-ui/dist/` is now included in the wheel. `synthadoc web` previously returned 503 on pip-installed instances because the React dist was not packaged.\n- **CI sync checks** — PRs now fail if `synthadoc/data/obsidian-plugin/`, `synthadoc/data/web-ui/`, or `README-pypi.md` are out of sync with their source. All sync-check steps use `shell: bash` for Windows runner compatibility.\n- **`bump_version.py` extended** — now also updates `synthadoc/data/obsidian-plugin/manifest.json` alongside the source manifest, preventing version drift between the bundled copy and the plugin source.\n\n### v0.9.2\n\n- **`synthadoc/data/web-ui/dist/` packaged** — web UI dist files committed to the repo and bundled in the wheel via hatchling artifacts.\n- **`scripts/sync_web_ui.py`** — helper script to sync `web-ui/dist/` into `synthadoc/data/web-ui/dist/` after `npm run build`.\n\n### v0.9.1\n\n- **PyPI metadata** — added `readme`, `keywords`, `classifiers`, and `[project.urls]` to `pyproject.toml` for correct PyPI rendering.\n- **`README-pypi.md`** — generated from `README.md` by `scripts/prepare_pypi_readme.py`; strips `pypi-strip` blocks and rewrites relative links to absolute GitHub URLs.\n- **Branch protection compatibility** — removed post-merge `build-plugin` CI job that pushed compiled plugin directly to `main`. Plugin sync is now enforced in PRs: CI fails if `synthadoc/data/obsidian-plugin/` is out of sync.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 5). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log/history/rollback`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n"}, "1853d175f2ac97c64419825bdc7e33f1b595265f": {"t": 1785767241.3084006, "n": 225387, "h": "d0d237a0fb2b22a7", "c": "# Synthadoc — Design Document\n\n**Version:** 1.1.3 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n29. [Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer)\n30. [Per-Source Truncation Flag](#30-per-source-truncation-flag)\n31. [Proportional Context Budget](#31-proportional-context-budget)\n32. [Knowledge Graph](#32-knowledge-graph)\n33. [Multi-Platform Agent Skill Files](#33-multi-platform-agent-skill-files)\n34. [Synthadoc Scaffold Marker](#34-synthadoc-scaffold-marker)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for Codex CLI, OpenCode, and generic OpenAI agents\n CLAUDE.md ← same content as AGENTS.md; loaded automatically by Claude Code\n GEMINI.md ← same content as AGENTS.md; loaded automatically by Gemini CLI\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the full source text (bounded by `max_source_chars`) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `flag` / `skip`). Pages with `status='active'` are protected — sources that conflict with them trigger `flag` rather than `update` (RULE 1b).\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Orchestrator calls `_guard_ingest_cost()` on the returned cost: soft warn → `logger.warning` only; hard gate → `cost_gate_exceeded` audit event + `fail_permanent` (job → DEAD). On success, job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 0b — Key Data Extraction | None (regex) | Deterministic pre-processor: extracts numbers, percentages, currency amounts, ratios, formulas, underscore identifiers, and date ranges from source text via regex; appends them as a `[Key Data — extracted by pre-processor]` section before any LLM sees the text. Zero token cost; cannot hallucinate. Anchors the synthesis LLM to all quantitative candidates. |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads the full source text (bounded by `max_source_chars`) + BM25 candidates + `purpose.md` scope + status of each candidate page. Outputs per-page action: `create`, `update`, `flag`, `skip`. **RULE 1b:** a page with `status='active'` is human-reviewed and authoritative — when the source provides a conflicting value, date, formula, or conclusion, the action must be `flag`, not `update`. The only exception is a source that adds a topic the page does not yet mention at all. Decision cache version: `v2`. |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. If no citation markers are present in the returned body, a `citation_pass4_no_markers` WARNING audit event is written — the page is still saved but the condition is flagged for re-ingest with a more capable model. Zero-citation results are not cached so a subsequent re-ingest can succeed. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic. **Suppressed when TF fallback was used** (a small corpus with broad coverage may legitimately return fewer than 3 candidates — that is expected, not a gap).\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n| Citation presence (Check 5b) | Page body has ≥ 50 words and zero `^[filename:L-L]` citation markers → WARNING. Diagnostic only — does not block promotion to `active`. Indicates the annotation pass failed to produce markers, usually a model-compatibility issue (use Gemini 2.5 Flash or higher for reliable citation annotation). |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n### ActionAgent\n\nDispatches action-intent queries from the chat UI (e.g. \"activate a draft page\", \"show lint report\", \"archive a stale page\", \"run scaffold\"). Parses the user's free-text intent into a structured action via an LLM extraction prompt, then executes the action against the wiki and returns a human-readable result. When the intent is ambiguous (e.g. \"activate a page\" without specifying which), sets `needs_clarification=True` and returns a prompt and candidate list for the web UI to render as chip buttons.\n\n### ScaffoldAgent\n\nGenerates and updates the wiki's structural pages. Reads all current wiki pages and asks the LLM to organise them into 5–8 domain-appropriate categories, then writes `wiki/index.md` (category index with wikilinks), `wiki/purpose.md` (scope statement), and `AGENTS.md` (ingest and query guidelines). Also stamps a `categories:` field on each page's YAML frontmatter. When `ROUTING.md` already exists, regenerates it to stay in sync with the updated index structure.\n\n### RewriteAgent\n\nRewrites follow-up questions in multi-turn conversations into self-contained, standalone form before BM25 retrieval. Converts context-dependent phrases (\"What came after that?\", \"Tell me more about his early life\") into explicit questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the correct pages without relying on conversation context. Only invoked when conversation history is non-empty. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### SummarizeAgent\n\nCompresses the oldest conversation turns into a single `[Session summary: …]` assistant turn when a session exceeds `conversation_history_turns`. Prevents unbounded context growth across long sessions. Emits a `notice` SSE event the first time compression occurs so the user can see that earlier context was condensed. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### ContextAgent\n\nBuilds token-budgeted context packs for MCP tool consumers. See [Section 20 — Context Packs](#20-context-packs) for full detail.\n\n### SearchDecomposeAgent\n\nDecomposes a web search intent into 1–4 terse keyword search strings optimised for authoritative source retrieval. Uses a separate prompt from `QueryAgent`'s question decomposition — search decomposition asks \"what distinct search strings would find the best sources?\" (terse keyword phrases) rather than \"what distinct questions does this ask?\" (natural-language sub-questions). See [Web search fan-out](#web-search-fan-out) in the IngestAgent section.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n| `session` | `.jsonl` | `claude session`, `codex session`, `cursor session`, `ai session`, `session history` | Extracts human-readable turns from AI coding session transcripts. Supports Claude Code JSONL format and Codex/Cursor format. Filters tool calls, thinking blocks, and sub-agent scaffolding. Short turns (< 20 assistant words, < 3 user words) are skipped. No external dependencies. |\n\n### Session Skill — AI Session History Ingestion\n\nThe `session` skill turns AI coding session history files (`.jsonl`) into searchable wiki pages capturing the problem-solving knowledge exchanged during the session.\n\n**Supported formats**\n\n| Format | File origin | Detection |\n|--------|-------------|-----------|\n| Claude Code | `~/.claude/projects//.jsonl` | `obj.type` ∈ `{\"user\", \"assistant\"}` with nested `obj.message` |\n| Codex / Cursor | Export from OpenAI Codex or Cursor IDE | `obj.role` at top level, no `message` wrapper |\n\nFormat is auto-detected from the first 30 parseable lines. Files that match neither format are parsed as Codex (fallback).\n\n**Filtering rules**\n\n| Content | Kept? | Reason |\n|---------|-------|--------|\n| User text ≥ 3 words | ✓ | Substantive question or instruction |\n| Assistant text ≥ 20 words | ✓ | Substantive answer |\n| User text < 3 words (`\"ok\"`, `\"yes\"`) | ✗ | Too terse to be useful |\n| Assistant text < 20 words | ✗ | Acknowledgement or one-liner |\n| `isSidechain: true` lines | ✗ | Sub-agent scaffolding (not the user's conversation) |\n| `tool_use` blocks | ✗ | Shell commands / file writes — not final output |\n| `tool_result` blocks | ✗ | Raw output — avoids leaking file contents or credentials |\n| `thinking` blocks | ✗ | Internal reasoning — not the final answer |\n| `permission-mode`, `system`, `last-prompt` lines | ✗ | Session metadata, not conversation |\n\nAfter extraction the text passes through Synthadoc's standard pre-LLM source sanitizer\n(zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars,\ninstruction-override phrases) — the same step applied to every PDF, DOCX, URL, and other source type.\nSee [§29 Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer).\n\n**Output format**\n\nEach kept turn is labelled and separated by `---`:\n\n```\n[USER]\nHow do I implement a sliding window algorithm in Python?\n\n---\n\n[ASSISTANT]\nA sliding window algorithm maintains a contiguous subarray by advancing two\npointers simultaneously…\n```\n\n**`suggested_slug`**\n\nThe skill sets `metadata[\"suggested_slug\"]` to `session-YYYY-MM-DD-` where the date comes from the file's `mtime` and the topic from the first substantive user turn (first 6 words, slugified). Example: `session-2026-07-15-how-do-i-implement-sliding`.\n\n**Limitations**\n\n- No chunking in v1.1 — very long sessions are truncated at `max_source_chars` (default 400 000 chars). Split large archives into individual `.jsonl` files.\n- Tool output is excluded by design — re-ingest the original source files if you need the file contents in the wiki.\n- Re-ingesting the same `.jsonl` file uses Synthadoc's standard source-hash dedup — no duplicate pages are created.\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n| `content_snapshot` | TEXT | Full page body at transition; NULL for lint/ingest events |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n**Compound identifier expansion:** When a token contains underscores, the tokenizer emits both the compound form and each component as separate tokens (`capex_growth` → `[\"capex_growth\", \"capex\", \"growth\"]`). This expansion is applied at both index time and query time so that human-friendly queries match programmatic identifiers and vice versa. Leading/trailing underscores are stripped before splitting; no empty strings are emitted.\n\n**TF fallback for small corpora:** When all BM25 scores in a result set are zero or negative (IDF collapse on a corpus of ≤ 2 documents, or a term present in every page), the search system falls back to normalised term frequency (TF) scoring: `TF(doc, query) = Σcount(term, doc) / len(doc_tokens)`. Results are returned in the same format as BM25 results with a `tf_fallback: True` flag on each `SearchResult`. This field is recorded in the audit trail and consumed by gap detection (see below).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp, cascade_links_removed_from: [str]}` _(cascade field added v1.0.2)_ |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n| `GET` | `/graph` _(v1.0.0)_ | — | `{status, node_count, edge_count, cluster_count, nodes: [...], edges: [...]}` or `{status: \"computing\"}` on first call |\n| `GET` | `/pages/{slug}/history` | `?index=N&include_content=true` | List content snapshots; ?index=N&include_content=true for single |\n| `POST` | `/pages/{slug}/rollback` | `{index: int, reason: str}` | Restore page body to snapshot N |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n**External file paths** (outside the wiki root) are supported when the request comes from\n`127.0.0.1` or `::1` and the payload includes `\"allow_external_paths\": true`. The CLI sets\nthis flag automatically for all local file sources. Remote clients cannot set this flag\n(the server ignores it for non-localhost connections) to prevent arbitrary file reads\nin server-exposed deployments.\n\nThis is required for ingesting files that live outside the wiki directory by design —\nfor example, Claude Code session transcripts at `~/.claude/projects//.jsonl`.\n\n**Client behaviour summary:**\n\n| Client | Sends `allow_external_paths` | Can ingest outside wiki root? |\n|--------|------------------------------|-------------------------------|\n| CLI (`synthadoc ingest `) | `true` (automatic, local file paths only) | Yes — server is localhost |\n| Obsidian plugin | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Web UI | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Direct HTTP API (localhost) | Set manually in request body | Yes — if request from 127.0.0.1/::1 |\n| Direct HTTP API (remote) | Ignored by server | No — server silently treats as `false` |\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 1.1.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n**Reading View default:** `synthadoc plugin install` and `synthadoc plugin upgrade` write `\"defaultViewMode\": \"preview\"` to `.obsidian/app.json` after copying the plugin files. This causes Obsidian to open new notes in Reading View by default, which is required for `^[filename:L-L]` citation chips to render. Only `defaultViewMode` is written; all other keys in `app.json` are preserved. If `app.json` is malformed JSON the write is skipped and a warning is logged — the install still succeeds.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports two things: dangling slugs (pages listed in ROUTING.md that no longer exist in the wiki) and unassigned slugs (pages that exist in index.md but are not listed in any ROUTING.md branch). Enabled only when ROUTING.md exists. **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n| `Graph: show knowledge graph` _(v1.1.0)_ | Open the knowledge graph panel — draggable modal with Canvas force-directed graph, type filter dropdown, cluster legend, hover tooltip showing title/slug/type/state/cluster/connections, and click-to-open page in current pane. System pages (index, overview, dashboard, purpose, log) are excluded. Capped at 300 most-connected nodes for large wikis with an explanatory banner. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [-w wiki] — generate ROUTING.md from index.md branch structure\n│ ├── validate [-w wiki] — report dangling slugs and cross-branch duplicates\n│ └── clean [-w wiki] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [-w wiki]\n├── candidates\n│ ├── list [-w wiki]\n│ ├── promote |--all [-w wiki]\n│ └── discard |--all [-w wiki]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ ├── log [slug] [-w wiki] [--state ]\n│ ├── history [-w wiki] [--index N] [--show-content] list content snapshots (newest first)\n│ └── rollback [-w wiki] [--index N] [--reason \"\"] restore page body to a snapshot\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** subscription, or want to use the free **Opencode Zen** tier, you can use either as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode Zen (free, no API key — requires connecting first: run `opencode` → `/connect` → select Zen):\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"opencode/big-pickle\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 30\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n# Citation pass (Pass 4) tuning — these two settings work together:\n# citation_source_lines — how many lines of the source the LLM sees when placing ^[...] markers.\n# Increase if lint reports out_of_range on long sources (transcripts, PDFs).\n# citation_max_tokens — output token budget for the annotated section returned by the LLM.\n# Increase if you raise citation_source_lines and have long wiki sections.\n# citation_source_lines = 400\n# citation_max_tokens = 8192\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax`, `qwen` (DashScope), and `deepseek` providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `30` | Exponential backoff base; delays are `min(base × 2^(attempt−1), 300)` seconds |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Emit `logger.warning` in server log when per-job ingest cost exceeds this threshold; job continues normally |\n| `cost.hard_gate_usd` | float | `2.00` | Permanently fail the ingest job (DEAD) and record a `cost_gate_exceeded` audit event when per-job cost exceeds this threshold |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × (backup_count + 1)` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n| `ingest.max_source_chars` | int | `32000` | Character limit applied to each source before the LLM call. Sources exceeding this limit are truncated; the page's `sources:` frontmatter entry gets `truncated: true` and lint emits a warning. Override per-run with `--max-source-chars N`. _(v1.0.0)_ |\n| `ingest.citation_source_lines` | int | `400` | Number of source lines the LLM sees during Pass 4 (citation annotation). Increase if lint reports `out_of_range` on long sources such as transcripts or PDFs. Raise `citation_max_tokens` in proportion when increasing this value. |\n| `ingest.citation_max_tokens` | int | `8192` | Output token budget for the annotated section returned by Pass 4. Increase when `citation_source_lines` is raised or wiki sections are long, to avoid truncated annotations. |\n| `query.context_wiki_pct` | float | `0.60` | Fraction of the model context window reserved for wiki source pages. _(v1.0.0)_ |\n| `query.context_history_pct` | float | `0.20` | Fraction reserved for conversation history. _(v1.0.0)_ |\n| `query.context_system_pct` | float | `0.15` | Fraction reserved for system prompt and instructions. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_index_pct` | float | `0.05` | Fraction reserved for the wiki index preamble. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_window` | int | _(auto)_ | Override the built-in context window lookup for the configured model. Use when running a local or custom model whose window size is not in the built-in table. _(v1.0.0)_ |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement is wired into the ingest pipeline: `Orchestrator._guard_ingest_cost()` is called after each ingest LLM call with the returned cost, before the job is marked complete.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | `logger.warning` in server log; job continues normally |\n| `hard_gate_usd` | $2.00 | Server/ingest path: records `cost_gate_exceeded` audit event + permanently fails job (DEAD). Interactive CLI: prompts `Proceed? [y/N]` |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `min(backoff_base_seconds × 2^(attempt − 1), 300)` \nWith the default base of 30 s: attempt 1 waits 30 s, attempt 2 waits 60 s, attempt 3 waits 120 s (all capped at 300 s). The `retry_after` timestamp is stored in the jobs table; `dequeue()` skips any job whose `retry_after` is still in the future. Applied only to transient errors (network timeouts, connection failures, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` by default. The bind address is configurable via `server.host` in `config.toml` (e.g. `host = \"0.0.0.0\"` to expose on all interfaces for LAN access). No built-in authentication — restrict via firewall when exposing beyond loopback. Remote access without a reverse proxy is not recommended on shared or networked machines.\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs and unassigned slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `-w ` / `--wiki ` to target a specific wiki.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; when all local source files are missing from disk the page is transitioned to `archived` (if only some sources are missing it is transitioned to `stale` instead). Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: lint evaluates all local (non-URL) source paths listed in the page's frontmatter. If **all** of them are missing from disk, the page is transitioned to `archived`. If only **some** are missing — meaning valid content from at least one source still exists — the page is transitioned to `stale` instead, preserving it for re-ingest rather than discarding it. _(v1.0.2: previously any single missing source triggered an immediate archive.)_\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Cascade link cleanup\n\nWhenever a page transitions to `archived` — whether triggered manually (`synthadoc lifecycle archive`), via the Obsidian Lifecycle modal, the MCP `synthadoc_lifecycle` tool, or automatically by lint detecting a missing source — Synthadoc immediately scans all other active pages and removes every `[[archived-slug]]` wikilink pointing to the now-archived page.\n\n- **Inline links** — `[[archived-slug]]` becomes plain text (the display text if an alias was used, otherwise the slug itself).\n- **List-item links** — a list item whose only content is `[[archived-slug]]` is dropped entirely.\n- **Archived pages are skipped** — pages already archived are not rewritten.\n- **System pages are skipped** — index, overview, dashboard, log, and purpose pages are excluded.\n\nThe cleanup runs synchronously before the response is returned, so the wiki is consistent immediately. The CLI and MCP responses include a `cascade_links_removed_from` field listing the affected slugs. During `lint run`, cascade is batched once after all auto-archive transitions complete, and the count is added to `dangling_links_removed` in the lint report.\n\nLint's existing periodic dangling-link cleanup (`lint run --scope orphans`) remains in place as a safety net for any dead links that predate this feature or arrive through other paths.\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc lifecycle history -w [--index ] [--show-content] [--json]\n List content snapshots captured at lifecycle transitions (newest first).\n --index inspects a single snapshot; --show-content prints the full body.\n\nsynthadoc lifecycle rollback -w --index --reason \"...\"\n Restore the page body from snapshot . The current body is saved as a new\n snapshot first, so every rollback is undoable.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n### Page Content Snapshots\n\nWhen the user activates, archives, or restores a page via the CLI, MCP tool, or REST\nAPI, the page body (`WikiPage.content`) is captured in the `content_snapshot` column of\nthe triggering `lifecycle_events` row.\n\nLint-agent and ingest-agent transitions do not capture snapshots — they do not change\npage content.\n\nSnapshots are indexed 1-based, newest first. Index 1 always refers to the most recent\ntransition that has a snapshot. The rollback operation:\n\n1. Records the current body as a new snapshot (making rollback undoable).\n2. Writes the target snapshot body to the `.md` file on disk.\n3. Does **not** change `page_states` or call `set_page_state`.\n\nStorage: ~3 KB per snapshot. Existing `purge_lifecycle_events()` removes rows\n(and their snapshots) on `--before` or `--keep-latest` purge.\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [...], \"input_tokens\": 1240, \"output_tokens\": 380, \"tokens_used\": 1620}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […], \"input_tokens\": N, \"output_tokens\": N, \"tokens_used\": N}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Streaming Token Audit\n\nStreaming LLM responses do not return token counts in the same way as blocking calls. To track real usage in the audit trail:\n\n**Provider-side collection** — each `LLMProvider` subclass exposes two instance attributes: `last_stream_input_tokens` and `last_stream_output_tokens`. These default to `0` and are populated from the API-specific usage field when the stream ends:\n\n| Provider | Mechanism | Verified |\n|---|---|---|\n| OpenAI | `stream_options={\"include_usage\": True}` passed to `chat.completions.create()`; final chunk carries `chunk.usage.prompt_tokens` / `completion_tokens` with empty `choices` | ✅ |\n| Anthropic | `message_start` event → `event.message.usage.input_tokens`; `message_delta` event → `event.usage.output_tokens` | ✅ |\n| Ollama | Final chunk with `done=True` → `prompt_eval_count` / `eval_count` | ✅ |\n| DeepSeek | Same `OpenAIProvider` path as OpenAI; DeepSeek's OpenAI-compatible API supports `stream_options` | ✅ |\n| MiniMax (reasoning: M2.5+) | Detects `` in stream → falls back to blocking `complete()` → captures exact counts from `resp.usage` | ✅ |\n| MiniMax (non-reasoning, e.g. M3 thinking=disabled) | Same `OpenAIProvider` path; MiniMax API silently ignores `stream_options`. Falls back to character-based estimate (÷ 3.5 chars/token) from prompt + answer lengths. Accuracy ±20%. | ✅ estimated |\n| Gemini | Same `OpenAIProvider` path via `generativelanguage.googleapis.com/v1beta/openai/`; Google's compatibility layer honours `stream_options` — verified live (Gemini 2.5 Flash Lite, 50K tokens) | ✅ |\n| Groq | Same `OpenAIProvider` path; Groq's OpenAI-compatible API supports `stream_options` | ✅ |\n| Qwen (DashScope) | Same `OpenAIProvider` path via DashScope; `stream_options` honoured — verified live (qwen-plus, 28K tokens) | ✅ |\n| Qwen (Ollama) | Ollama path → `prompt_eval_count` / `eval_count` | ✅ |\n\n**Forwarding to `done` event** — after `complete_stream()` is exhausted, `QueryAgent.run_stream()` reads the provider attributes and includes them in the final `done` event payload: `input_tokens`, `output_tokens`, `tokens_used`.\n\n**Audit record** — `Orchestrator.query_stream()` consumes the `done` event and calls `estimate_cost(model, input_tokens, output_tokens, is_local)` from `synthadoc/providers/pricing.py`, then calls `AuditDB.record_query()` with the real token count and computed cost. The audit row at `GET /audit/queries` will therefore always show non-zero values for cloud providers.\n\nWhen a provider does not report streaming usage (e.g. a future provider not yet wired), the attributes stay at `0`; the audit row records `tokens=0, cost_usd=0.0` rather than crashing, matching the pre-fix behaviour.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 5) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 5 # turns to include in each request (default: 5; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp, cascade_links_removed_from: [str]}` or `{error, cascade_links_removed_from: []}` _(cascade field added v1.0.2)_ | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions for Codex CLI / OpenCode (if present)\n├── CLAUDE.md ← same content; loaded by Claude Code (if present)\n├── GEMINI.md ← same content; loaded by Gemini CLI (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── hooks/ ← user hook scripts (always included)\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ ├── extracted/ ← text sidecars + PDF pagemaps (always included)\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `logs/`.\n\n> **v0.9.3 fix:** `hooks/` and `.synthadoc/extracted/` were previously missing from backups. Omitting `extracted/` caused the Source Viewer and Provenance citation modal to show \"Could not read source file\" after restore.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## 29. Pre-LLM Source Sanitizer\n\nEvery source document passes through a sanitizer immediately after text extraction, before any LLM call. The sanitizer removes six categories of content that could manipulate the LLM or degrade compilation quality:\n\n| Category | Action | Warning logged? |\n|---|---|---|\n| Zero-width characters (U+200B, U+200C, U+200D, U+FEFF) | Removed silently | No |\n| Bidi override characters (U+202A–U+202E, U+2066–U+2069) | Removed | Yes |\n| HTML comments (``) | Removed silently | No |\n| Hidden CSS spans (`display:none`, `visibility:hidden`) | Removed silently | No |\n| Base64 blobs ≥ 200 consecutive characters | Replaced with `[base64 content removed]` | Yes |\n| Instruction-override phrases (8 patterns: \"ignore previous instructions\", \"disregard the above\", \"override your system prompt\", etc.) | Replaced with `[redacted]` | Always |\n\nWhen content is removed and a warning is appropriate, the server logs a `WARN` entry:\n```\n[WARN] sanitizer stripped content from 'papers/survey.pdf': bidi overrides, instruction-override phrase\n```\n\nThis is a pure pre-processing step with zero LLM cost. Truncation (if configured) is applied after sanitization, so the character budget is consumed by clean text only.\n\n---\n\n## 30. Per-Source Truncation Flag\n\nLarge source documents are truncated before they reach the LLM. By default the limit is **32,000 characters** (~8,000 tokens), raised from the prior hardcoded 8,000-character limit. When a source exceeds this limit, its compiled wiki page records a `truncated: true` flag in its `sources:` frontmatter entry.\n\n### Configuration\n\n```toml\n[ingest]\nmax_source_chars = 32000 # default; raise for large PDFs or books\n```\n\n### Per-ingest override\n\nThe limit can be raised for a single ingest run without modifying the config:\n\n```bash\nsynthadoc ingest papers/large-textbook.pdf --max-source-chars 128000\n```\n\nThe same `max_source_chars` field is accepted by the HTTP `POST /jobs/ingest` body and the MCP `synthadoc_ingest` tool.\n\n### Lint surfacing\n\n`synthadoc lint` emits a warning for any page that has a truncated source:\n\n```\n[WARN] quantum-computing.md: source 'papers/large-survey.pdf' was truncated at ingest\n (source exceeded max_source_chars=32000 — 87,412 chars in source).\n To re-ingest with a higher limit (this source only):\n synthadoc ingest papers/large-survey.pdf --max-source-chars 128000\n To raise the limit for all future ingests:\n set [ingest] max_source_chars = 128000 in your config\n```\n\n---\n\n## 31. Proportional Context Budget\n\nQuery context is allocated proportionally to the configured model's context window, replacing the prior hard cap. The default allocation:\n\n| Slice | Default | Purpose |\n|---|---|---|\n| `context_wiki_pct` | 60% | Wiki page content (ranked by BM25 + vector score) |\n| `context_history_pct` | 20% | Chat turn history (newest-first, oldest dropped when over budget) |\n| `context_system_pct` | 15% | System prompt and purpose document |\n| `context_index_pct` | 5% | `ROUTING.md` and search index |\n\nPercentages must sum to ≤ 100. Configure in `[query]`:\n\n```toml\n[query]\ncontext_window = 0 # 0 = auto-detect from known model map\ncontext_wiki_pct = 60\ncontext_history_pct = 20\ncontext_system_pct = 15\ncontext_index_pct = 5\n```\n\n> **Note:** `context_system_pct` and `context_index_pct` are parsed and validated but not yet enforced as hard caps in this release — the system prompt uses a conservative fixed limit, which is well within the configured 15% slice for all supported models. Full per-slice enforcement is planned for v1.1.\n\n### Model context window map\n\nSynthadoc ships a built-in prefix-matched table:\n\n| Model | Context window |\n|---|---|\n| `claude-opus-4*`, `claude-sonnet-4*`, `claude-haiku-4*` | 200,000 tokens |\n| `gpt-4o*`, `gpt-4-turbo*` | 128,000 tokens |\n| `gpt-4` (exact) | 8,192 tokens |\n| `gpt-3.5-turbo*` | 16,385 tokens |\n| Unknown / fallback | 128,000 tokens |\n\nSet `context_window = N` in config to override the map (useful for local Ollama models with non-standard context sizes).\n\nAt the defaults on a 200k-token model: wiki slice ≈ 480,000 chars — far more than any practical query needs. On constrained models, the budget scales down automatically so the same config works across model families.\n\n---\n\n## 32. Knowledge Graph\n\nSynthadoc computes a wikilink graph across all active and draft pages during every lint run. The graph is stored in two tables in `audit.db`:\n\n```sql\nCREATE TABLE graph_nodes (\n slug TEXT PRIMARY KEY,\n cluster_id INTEGER NOT NULL,\n updated_at TEXT NOT NULL\n);\n\nCREATE TABLE graph_edges (\n from_slug TEXT NOT NULL,\n to_slug TEXT NOT NULL,\n weight INTEGER NOT NULL DEFAULT 1,\n edge_type TEXT NOT NULL DEFAULT 'mixed',\n PRIMARY KEY (from_slug, to_slug)\n);\n```\n\nEdge weight is a composite signal from two sources:\n\n| Edge type | Source | Weight contribution |\n|-----------|--------|---------------------|\n| `wikilink` | `[[slug]]` occurrences in page body | +1 per occurrence |\n| `co_source` | pages compiled from the same source file (matched by hash) | +2 per shared source |\n\nMost edges are `mixed` (have both types). Pure `co_source` edges surface hidden relationships — pages compiled from the same source document are linked immediately after ingest, before any wikilinks are created. `edge_type` is stored on each edge so the web UI can render co-source edges with a different visual style.\n\nCluster IDs are assigned by Louvain community detection — pages that link densely to each other end up in the same cluster.\n\n### REST API\n\n**`GET /graph`** — returns the full graph with per-node enrichment:\n\n```json\n{\n \"status\": \"ready\",\n \"node_count\": 45,\n \"edge_count\": 123,\n \"cluster_count\": 4,\n \"nodes\": [\n {\"slug\": \"quantum-error-correction\", \"title\": \"Quantum Error Correction\",\n \"type\": \"concept\", \"state\": \"active\", \"cluster_id\": 2}\n ],\n \"edges\": [\n {\"from\": \"quantum-error-correction\", \"to\": \"shor-algorithm\", \"weight\": 3, \"edge_type\": \"mixed\"}\n ]\n}\n```\n\nWhen the graph tables are empty (first call after upgrade, or before the first lint run), the server returns `{\"status\": \"computing\"}` and starts a background build immediately. The web UI polls every 2 seconds and renders the graph when the status becomes `\"ready\"`.\n\n### Web UI\n\nThe **Graph** tab (alongside Chat in the top nav) renders a D3.js force-directed graph. Nodes are colored by cluster. Edge thickness is proportional to weight (1–4 px). Pure `co_source` edges (no wikilink) are rendered with a dashed stroke, making hidden source-based relationships visually distinct from explicit cross-links. Clicking a node opens a sidebar showing the page title, type and lifecycle state badges, cluster assignment, and an **\"Ask about this →\"** button that switches to the Chat tab with a pre-filled query.\n\nControls: zoom in/out (scroll or pinch), drag nodes, filter by page type.\n\n### System page exclusion _(v1.1.0)_\n\nThe five structural system pages — `overview`, `index`, `dashboard`, `purpose`, and `log` — are excluded from both `graph_nodes` and `graph_edges`. These pages link to nearly every other page and would act as false hubs, inflating edge counts and distorting Louvain cluster detection. System page exclusion is applied inside `_build_graph()` before any edge computation.\n\n### Obsidian knowledge graph panel _(v1.1.0)_\n\n→ Full description: [§33 Multi-Platform Agent Skill Files](#33-multi-platform-agent-skill-files) covers scaffold; the graph panel itself is documented in [§8 Obsidian Plugin](#8-obsidian-plugin) command table.\n\nThe Obsidian plugin includes a **draggable modal** knowledge graph panel (`Graph: show knowledge graph` command). It fetches from the same `GET /graph` endpoint as the web UI and renders a force-directed Canvas 2D simulation using Verlet integration (no external library). System pages are already excluded from the server response.\n\n| Feature | Obsidian panel | Web UI Graph tab |\n|---------|----------------|-----------------|\n| Engine | Canvas 2D + Verlet | D3.js SVG |\n| Node click | Opens the wiki page in the current pane | Opens the node sidebar with \"Ask about this →\" |\n| **Best for** | **Navigating to and editing pages** | **Asking questions about a page** |\n| Draggable panel | Yes — drag header to reposition | Fixed tab layout |\n| Node cap | 300 most-connected (see note) | No cap |\n| Cluster legend | Yes | Yes |\n| Type filter | Yes | Yes |\n\n**Rule of thumb:** Use the **web UI** when you want to ask questions about a node. Use the **Obsidian panel** when you want to navigate to and edit a page.\n\n**Node cap rationale:** The Obsidian panel uses a hand-written Verlet integrator with O(n²) pairwise charge repulsion — every node repels every other node on each animation tick. At 300 nodes that is ~90,000 operations per frame, which runs smoothly at 60 fps inside Obsidian's Electron renderer. At 500 nodes it is ~250,000 operations per frame and starts to lag noticeably. The web UI has no cap because D3's `forceManyBody` uses a Barnes-Hut tree approximation (O(n log n)), which scales to thousands of nodes without frame-rate issues. When the cap is hit, nodes are ranked by degree and the 300 most-connected are shown; a banner reports the total page count and the number of hidden nodes, and suggests using the type filter to bring the set below 300.\n\n→ User walkthrough: [Quick-Start Guide §24 — Knowledge Graph](docs/user-quick-start-guide.md#knowledge-graph)\n\n---\n\n## 33. Multi-Platform Agent Skill Files\n\n\n\n`synthadoc init` and `synthadoc scaffold` write three agent skill files at the wiki root:\n\n| File | Format | Read by |\n|------|--------|---------|\n| `AGENTS.md` | OpenAI Agents SDK / Codex / OpenCode | Codex CLI, OpenCode, generic agents |\n| `CLAUDE.md` | Claude Code / Anthropic | Claude Code (loaded automatically when the wiki folder is opened) |\n| `GEMINI.md` | Gemini CLI | Gemini CLI |\n\nAll three files carry **identical content**: a title block, the LLM-generated domain guidelines for the wiki, a quick-reference CLI command table, and an MCP tool table. The only difference is the heading comment. This means any AI coding tool that opens the wiki root gets full Synthadoc guidance without manual setup.\n\n### Content generated by `scaffold`\n\nEach skill file is generated from the same LLM call that produces `purpose.md` and `index.md`. The LLM receives the current wiki page list and domain description, then generates:\n\n1. **Domain label** — a precise human-readable name inferred from wiki content (e.g. `History of Computing`, not `My Wiki`)\n2. **Domain guidelines** — 3 domain-specific ingest and query guidelines as plain sentences\n3. **Quick Reference table** — pre-filled with the current wiki name and port\n\nIf the wiki was created before v1.1.0, running `synthadoc scaffold` once regenerates all three files and adds `CLAUDE.md` and `GEMINI.md` to any wiki that only had `AGENTS.md`.\n\n### Template source\n\nThe instruction body is defined in `synthadoc/cli/_init.py` as `_AGENT_INSTRUCTION_BODY` and shared by all three templates, keeping the files permanently in sync. The scaffold agent substitutes `{domain}`, `{guidelines}`, and `{port}` at write time.\n\n---\n\n## 34. Synthadoc Scaffold Marker\n\n\n\nThe scaffold marker `` (written as an HTML comment) separates **user-authored content** (preserved) from **scaffold-managed content** (rewritten on each `synthadoc scaffold` run).\n\n### Single-marker mode (index.md)\n\nThe marker appears once, below the H1 heading in `index.md`. Everything above the marker is the user zone; everything below is replaced with new LLM-generated categories.\n\n```markdown\nMy custom introduction — maintained by hand.\n\n\n\n## Pioneers and Visionaries\n- [[alan-turing]]\n...\n```\n\nIf the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### Per-section multi-marker mode (purpose.md)\n\n`purpose.md` uses a **per-section** approach: the marker appears inside each `## Section` heading. Within a section, content the user writes **above** the marker is preserved; content below is replaced with fresh LLM output. Sections without a marker in the existing file are replaced entirely (and a marker is added). Sections in the existing file that the scaffold template does not include are kept unchanged.\n\n```markdown\n## Overview\n\nMy custom overview note.\n\n\n\nLLM-generated overview content goes here.\n\n## What Belongs\n\n\n\nLLM-generated scope definition.\n```\n\nThis design lets users annotate any section of `purpose.md` without losing their notes on the next scaffold run.\n\n### Marker preservation guarantee\n\n- `preserve_user_zone()` in `scaffold_agent.py` handles both modes.\n- If a section has no marker in the existing file, `preserve_user_zone()` treats the entire section body as user content and preserves it, replacing only what falls below any marker it finds.\n- The scaffold NEVER touches `config.toml`, `dashboard.md`, or any wiki page (`wiki/*.md`).\n\n→ User walkthrough: [Quick-Start Guide §14 — Enrich the wiki with scaffold](docs/user-quick-start-guide.md#scaffold)\n\n---\n\n## Customization\n\nSynthadoc exposes four extension points — all hot-loaded, no server restart required.\n\n### Custom skills (new file formats)\n\nSubclass `BaseSkill` (Apache-2.0 — no AGPL obligation on your skill code), drop the folder in `/skills/` or `~/.synthadoc/skills/`, and Synthadoc picks it up on the next ingest. Skills match by file extension or intent prefix and support any Unicode text, including CJK prefixes.\n\n→ Full interface, manifest format, and examples: [§5 Skills System](#5-skills-system) and [§17 Plugin Development Guide](#17-plugin-development-guide)\n\n### Custom LLM providers\n\nSubclass `LLMProvider` from `synthadoc/providers/base.py` (Apache-2.0) and place the file in `~/.synthadoc/providers/` or the wiki `providers/` directory. Switch active provider with one config line.\n\n→ Full interface and wiring instructions: [§10 Configuration — Provider switching](#10-configuration)\n\n### Hooks\n\nShell scripts (any language) that fire on `on_ingest_complete` and `on_lint_complete`. Receive a JSON context on stdin. Set `blocking = true` to gate the operation on the hook's exit code — useful for CI pipelines and quality gates.\n\n→ Full event schema, context JSON, and blocking behaviour: [§11 Hook System](#11-hook-system)\n\n### Cache control\n\nThree cache layers (embedding, LLM response, provider prompt cache) invalidate automatically on source-file change (SHA-256). Force a fresh call with `--force`, or wipe all cached responses with `synthadoc cache clear -w `.\n\n→ Layer-by-layer breakdown and invalidation rules: [§12 Cache System](#12-cache-system)\n\n### Multi-Platform Agent Skill Files\n\nEvery Synthadoc wiki ships three companion files that give AI coding agents a complete, self-contained operating guide for the wiki:\n\n| File | Platform |\n|---|---|\n| `AGENTS.md` | Codex CLI, OpenCode, and any agent that reads the AGENTS.md convention |\n| `CLAUDE.md` | Claude Code (this session's tool) — highest-priority instruction source in Claude's hierarchy |\n| `GEMINI.md` | Gemini CLI — relevant because Synthadoc's default LLM provider is `gemini-2.5-flash-lite` |\n\nAll three files share identical body content generated from the same template; they differ only in their H1 heading so each agent recognises its own file first. The body covers:\n\n- **Domain guidelines** — LLM-generated bullet list of domain-specific ingest and query rules (produced by `scaffold`)\n- **Quick reference table** — every common CLI command with the correct `synthadoc` syntax\n- **Server startup** — how to start the server and verify it is running\n- **Ingest examples** — local file, URL, YouTube, agent session `.jsonl`, force reingest, dry-run\n- **Query** — basic and streaming query syntax; citation marker format\n- **Lint** — when to run and what each check does\n- **Lifecycle** — transition commands with state names\n- **Page schema** — frontmatter structure with all fields\n- **MCP tools table** — all 12 tools with purpose descriptions\n\n**How the files are created and updated:**\n\n1. `synthadoc init` — writes all three files with default domain guidelines and the wiki's configured port.\n2. `synthadoc scaffold` — regenerates all three with LLM-produced domain guidelines derived from the current wiki state. The `scaffold` command prints a confirmation line for each file.\n\n**Use cases:**\n\n- Open Claude Code in a Synthadoc wiki directory and it immediately knows how to ingest, query, lint, and manage lifecycle — no additional setup. Claude Code can also start the server in background mode (`synthadoc serve -w -b`), stop it cleanly, and orchestrate multi-step workflows, effectively acting as a control plane for the wiki.\n- Gemini CLI reads `GEMINI.md` and can run `synthadoc` commands with the correct flags without asking the user for help.\n- CI pipelines with Codex CLI use `AGENTS.md` to drive automated lint and lifecycle transitions on PR merge.\n- Agent session files produced by Claude Code (`.jsonl` under `~/.claude/projects/`) can themselves be ingested into the wiki, capturing the reasoning behind design decisions alongside the source documents.\n\n→ See [README.md — Interfaces & Integration](../README.md#interfaces--integration) for the feature comparison row and quick-start instructions.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v1.1.2\n\n- **Exponential backoff on transient URL fetch failures** — jobs failing with transient errors (connection timeouts, 5xx responses, dropped TCP connections including `RemoteProtocolError`) now re-queue with exponential backoff: `min(base × 2^(attempt−1), 300)` seconds (default base 30 s). Previously all failures retried immediately, exhausting the retry budget within seconds. `dequeue()` skips any job whose `retry_after` timestamp is still in the future; other jobs run normally during the wait window. See [§14 Job Queue](#14-job-queue).\n- **Accurate retry outcome in job warning logs** — the transient-failure warning now reads the current attempt count from the database before logging, so it correctly reports `will retry` vs. `final attempt` (was always `will retry` regardless of retry state).\n- **Scheduler subprocess timeout** — scheduled jobs (`lint`, `ingest --batch`, etc.) are now guarded by `asyncio.wait_for(proc.communicate(), timeout=job_timeout_seconds)`; a hung subprocess is killed after the timeout and the run is recorded as `failed`. Previously a stalled subprocess could block the scheduler task indefinitely. `job_timeout_seconds` defaults to 600 s and is now shown (uncommented) in fresh-init `config.toml`.\n- **Streaming query token and cost accuracy** — the streaming query audit record was written with `tokens=0, cost_usd=0.0`; it now records the actual LLM token usage and cost from the streamed response.\n- **Cost hard gate timing** — the cost hard gate now fires before LLM tokens are spent and pages are written; previously the gate ran after the LLM call completed, allowing partial page writes before rejection.\n- **Session state memory growth bounded** — `_session_state` in the HTTP server is pruned at 100 concurrent sessions (LRU eviction) to prevent unbounded memory growth on long-running servers.\n- **Streamed `` block cleanup** — a leaked fallback asyncio task is now cancelled when a streaming response ends inside an unclosed `` block (e.g. reasoning-model responses cut off mid-think).\n- **Config validation for staging fields** — `staging_policy` values other than `off`, `all`, and `threshold`, and non-numeric `staging_confidence_min`, were silently accepted and caused runtime errors; they now raise a `ConfigError` at server startup.\n- **Job status guard on retry** — retrying a job that is already `in_progress`, `done`, or `skipped` now returns an error instead of silently resetting it to `pending`.\n- **Jobs endpoint performance** — `GET /jobs/{id}` and `DELETE /jobs/{id}` now use a direct primary-key lookup instead of a full table scan; eliminates O(N) reads on large job queues.\n- **Vector migration GC safety** — the background vector migration `asyncio.Task` is now held in a strong reference to prevent premature garbage collection mid-run.\n- **Quota error stack trace suppression** — Qwen 403 errors and Claude Code session-limit exhaustion now emit a concise warning naming the actionable config key, instead of a full stack trace.\n- **Scaffold agent JSON robustness** — `MiniMax-M3` and other reasoning models can truncate mid-JSON; the scaffold agent now handles truncated JSON output gracefully rather than failing the job.\n- **Staging/Candidates section in agent skill files** — `AGENTS.md`, `CLAUDE.md`, and `GEMINI.md` generated by `synthadoc scaffold` now include a `## Staging and Candidates` section.\n- **Citation failure pagination fix** — `GET /audit/citation-failures` was applying the `LIMIT` in SQL before filtering by `page_slug`, returning incorrect results for paginated per-page queries; filtering now runs before the limit.\n\n### v1.1.0\n\n- **Weighted knowledge graph edges** — graph edges now carry two signals: wikilink occurrences (+1 each) and co-source connections — pages compiled from the same source file (matched by SHA-256 hash) — (+2 per shared source). `edge_type` field added to `graph_edges` table (`wikilink`, `co_source`, or `mixed`). `GET /graph` exposes `edge_type` per edge. Web UI renders edge thickness proportional to weight (1–4 px) and dashed lines for pure co-source edges. Co-source edges appear immediately after ingest — before any wikilinks — surfacing hidden relationships in the graph. Schema migrated to version 3.\n- **Multi-platform agent skill files** — `synthadoc init` and `synthadoc scaffold` write `CLAUDE.md`, `AGENTS.md`, and `GEMINI.md` at wiki root; all three carry LLM-generated domain guidelines and the full CLI quick-reference so Claude Code, Codex CLI, OpenCode, Pi, and Gemini CLI users get first-class guidance without manual setup. Run `synthadoc scaffold` once on any pre-v1.1.0 wiki to generate the new files. See [§33](#33-multi-platform-agent-skill-files).\n- **Session history ingestion skill** — `.jsonl` session files from Claude Code and Codex CLI are ingested as wiki pages; human turns and substantive assistant responses extracted; tool calls, thinking blocks, and sub-agent scaffolding skipped; format auto-detected; `suggested_slug` derived from session date and first user message. Converts ephemeral problem-solving knowledge into searchable wiki entries. See [§5 Skills System — Session Skill](#session-skill--ai-session-history-ingestion).\n- **Obsidian knowledge graph panel** — new `Graph: show knowledge graph` command in the Obsidian plugin opens a draggable modal with a Canvas 2D force-directed graph. Features: Verlet integration, type filter dropdown, cluster legend, hover tooltip (title, slug, type, state, cluster, connections), click-to-open page in current pane, node drag to repin. Closes only via the × button — Escape and outside-click do not close the panel, matching the behaviour of other Synthadoc modals. **Large-wiki cap:** when the node set exceeds 300 (after any type filter is applied), nodes are ranked by degree (total edge count) and the 300 most-connected are displayed; edges to capped-out nodes are removed; a banner explains the cap and suggests using the type filter to narrow the view below 300. See [§8 Obsidian Plugin](#8-obsidian-plugin) and [§32 Knowledge Graph — Obsidian panel](#obsidian-knowledge-graph-panel-v110).\n- **Per-section scaffold markers in `purpose.md`** — `purpose.md` now uses per-section `` markers. Users can write content above the marker in any `## Section` block; that content is preserved on every re-scaffold. A single-marker mode for `index.md` continues to work as before. See [§34 Synthadoc Scaffold Marker](#34-synthadoc-scaffold-marker).\n- **System pages excluded from knowledge graph** — `overview`, `index`, `dashboard`, `purpose`, and `log` are filtered out of `_build_graph()` before any edge computation so they cannot act as false hubs in Louvain cluster detection. Both the Obsidian graph panel and the web UI Graph tab reflect this exclusion.\n- **Web UI graph cluster legend** — the web UI Graph tab now shows a cluster legend below the canvas (matching the Obsidian plugin panel), with a color swatch and label for each Louvain cluster present in the filtered node set.\n- **`_merge_section_into_page` deduplication** — force re-ingest of the same source file (e.g. an updated `.jsonl` session) now replaces the page body rather than appending a duplicate section. `_files_match()` normalizes absolute vs. relative paths so the same file is recognized regardless of how it was passed to the ingest command.\n- **`purpose.md` domain label inference** — `scaffold` now infers a precise `domain_label` from current wiki content (e.g. `History of Computing`) instead of using a generic placeholder; the label is used in all three agent skill files and in `purpose.md`'s Overview section.\n- **CI: Windows runner fix** — `scripts/sync_plugin.py` and `scripts/sync_web_ui.py` now use `shutil.copy2` instead of `cp -f`, resolving CI failures on Windows runners for the sync check.\n\n### v1.0.2\n\n- **Cascade link cleanup on archive** — archiving any page immediately removes every `[[slug]]` wikilink pointing to it from all other active pages, without waiting for the next `lint run`. Inline links become plain text; list-item-only links are dropped entirely. The HTTP `POST /lifecycle/transition` and MCP `synthadoc_lifecycle` responses now include a `cascade_links_removed_from` field listing the rewritten page slugs. During `lint run`, cascade runs once after all auto-archive transitions complete and the count is included in the lint report's `dangling_links_removed` field. System pages (index, dashboard, purpose, log, overview) and already-archived pages are skipped. The existing dangling-link lint pass remains as a safety net for dead links that predate this release.\n- **Multi-source partial-archive fix** — when a page has multiple local source files and only some are missing from disk, lint now transitions the page to `stale` (not `archived`), preserving it for re-ingest. The page is only archived when every local source file is gone. URL sources are not affected by this check.\n\n### v1.0.1\n\n- **Scaffold JSON extraction hardened** — replaced the greedy regex used to extract the LLM's JSON response with a brace-balanced parser; eliminates false-positive matches on `[[wikilink]]` syntax that previously caused `json.loads()` failures and silent scaffold corruption\n- **Generated file validation** — scaffold now validates every LLM-generated file (index, purpose, AGENTS.md) for minimum length, required frontmatter, and `[[wikilink]]` presence before writing to disk; validation failure marks the job as failed with a structured error message rather than silently writing a malformed page\n- **ROUTING.md regeneration validation** — `routing init` and scheduled routing regeneration now validate the regenerated file for minimum line count and anchor presence before applying it; prevents a corrupt or truncated ROUTING.md from entering the wiki\n- **Scaffold failure reporting** — scaffold skip and failure reasons are now surfaced in the terminal output and job error field; previously silent failures (e.g. LLM timeout, JSON parse error) are now visible without tailing the JSON log\n- **Install-time scaffold removed** — `synthadoc install` no longer runs the LLM scaffold call during installation; scaffold is a separate step (`synthadoc scaffold`) that the user runs after setting their API key, avoiding confusing API-key-not-set errors at install time\n- **AquaFlow LLM evaluation** — 15-question M&A due-diligence benchmark across five models (MiniMax-Think M3, Claude Opus 4.8, Claude Sonnet 4.6, DeepSeek-R1, Qwen Plus); full report at `docs/example/aquaflow/evaluation/report/llm-query-benchmark.md`\n\n### v1.0.0\n\n- **Pre-LLM source sanitizer** — every source document passes through a sanitizer before any LLM call; removes six categories of potentially harmful content (zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars, instruction-override phrases); a `WARN` log entry is emitted when bidi overrides or instruction-override phrases are stripped; zero LLM cost; applied before truncation so the character budget is consumed by clean text only\n- **Per-source truncation flag** — default character limit raised from 8,000 to 32,000 characters per source (~8,000 tokens); truncated sources are flagged with `truncated: true` in the page's `sources:` frontmatter; `--max-source-chars N` CLI flag and matching `POST /jobs/ingest` body field and MCP `synthadoc_ingest` parameter override the limit per run; `synthadoc lint` emits a warning for any page with a truncated source and suggests the override command; configurable via `[ingest] max_source_chars` in `config.toml`\n- **Proportional context budget** — query context is allocated proportionally to the configured model's context window; four configurable slices: `context_wiki_pct` (60%), `context_history_pct` (20%), `context_system_pct` (15%), `context_index_pct` (5%); built-in prefix-matched model context window table covers Claude 4, GPT-4o/turbo/3.5, and GPT-4; unknown models fall back to 128,000 tokens; `context_window = N` in `[query]` overrides the table for local or custom models\n- **Knowledge graph** — wikilink graph computed across all active and draft pages during every lint run; stored in `graph_nodes` and `graph_edges` tables in `audit.db`; edge weight = number of `[[wikilink]]` references between two pages; cluster IDs assigned by Louvain community detection; `GET /graph` REST endpoint returns nodes with per-node enrichment (title, type, state, cluster) and edges; first call after upgrade triggers a background build and returns `{\"status\":\"computing\"}`; web UI **Graph** tab (D3.js force-directed graph) — nodes colored by cluster, click to see page details and an \"Ask about this →\" button that opens a pre-filled chat query\n- **Citation accuracy & rendering improvements** — six correctness fixes to the ingest citation pipeline: (1) full source text passed to the decision LLM so citations can reference any line in the document; (2) `_annotate_citations` bugs A–F resolved (false-positive matches, off-by-one line ranges, duplicate markers, escaped bracket handling, multi-paragraph span errors, and source filename normalisation); (3) `_has_citations` check added — lint warns and suggests a model upgrade when a page has zero citation markers after synthesis; (4) `Key Data` section extraction hardened so numerical facts, rates, and formulas are preserved verbatim across the annotation pass; (5) `obsidianCitationsToGfm` rendering fixed in the web UI — citation superscripts now display correctly in the chat panel alongside streamed answers; (6) reading view set as the default Obsidian display mode for wiki pages\n\n### v0.9.3\n\n- **Backup now includes `hooks/` and `.synthadoc/extracted/`** — both directories were silently omitted from backups. `hooks/` contains user-customised hook scripts; `.synthadoc/extracted/` contains per-source text sidecars and PDF pagemaps written by the ingest agent. Without these, the Source Viewer and Provenance citation modal showed \"Could not read source file\" after restore.\n- **PyPI publish workflow** — `publish.yml` triggers on GitHub Release, builds the wheel with hatchling, and uploads to PyPI via OIDC trusted publishing (no API token stored). `pip install synthadoc` now works.\n- **Web UI bundled in pip package** — `synthadoc/data/web-ui/dist/` is now included in the wheel. `synthadoc web` previously returned 503 on pip-installed instances because the React dist was not packaged.\n- **CI sync checks** — PRs now fail if `synthadoc/data/obsidian-plugin/`, `synthadoc/data/web-ui/`, or `README-pypi.md` are out of sync with their source. All sync-check steps use `shell: bash` for Windows runner compatibility.\n- **`bump_version.py` extended** — now also updates `synthadoc/data/obsidian-plugin/manifest.json` alongside the source manifest, preventing version drift between the bundled copy and the plugin source.\n\n### v0.9.2\n\n- **`synthadoc/data/web-ui/dist/` packaged** — web UI dist files committed to the repo and bundled in the wheel via hatchling artifacts.\n- **`scripts/sync_web_ui.py`** — helper script to sync `web-ui/dist/` into `synthadoc/data/web-ui/dist/` after `npm run build`.\n\n### v0.9.1\n\n- **PyPI metadata** — added `readme`, `keywords`, `classifiers`, and `[project.urls]` to `pyproject.toml` for correct PyPI rendering.\n- **`README-pypi.md`** — generated from `README.md` by `scripts/prepare_pypi_readme.py`; strips `pypi-strip` blocks and rewrites relative links to absolute GitHub URLs.\n- **Branch protection compatibility** — removed post-merge `build-plugin` CI job that pushed compiled plugin directly to `main`. Plugin sync is now enforced in PRs: CI fails if `synthadoc/data/obsidian-plugin/` is out of sync.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 5). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log/history/rollback`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n"}, "c1db6b4c243a91a332c8417d2b99ab3cee0a4bae": {"t": 1785767243.1334867, "n": 223563, "h": "bc852c53cf3bec37", "c": "# Synthadoc — Design Document\n\n**Version:** 1.1.3 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n29. [Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer)\n30. [Per-Source Truncation Flag](#30-per-source-truncation-flag)\n31. [Proportional Context Budget](#31-proportional-context-budget)\n32. [Knowledge Graph](#32-knowledge-graph)\n33. [Multi-Platform Agent Skill Files](#33-multi-platform-agent-skill-files)\n34. [Synthadoc Scaffold Marker](#34-synthadoc-scaffold-marker)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for Codex CLI, OpenCode, and generic OpenAI agents\n CLAUDE.md ← same content as AGENTS.md; loaded automatically by Claude Code\n GEMINI.md ← same content as AGENTS.md; loaded automatically by Gemini CLI\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the full source text (bounded by `max_source_chars`) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `flag` / `skip`). Pages with `status='active'` are protected — sources that conflict with them trigger `flag` rather than `update` (RULE 1b).\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Orchestrator calls `_guard_ingest_cost()` on the returned cost: soft warn → `logger.warning` only; hard gate → `cost_gate_exceeded` audit event + `fail_permanent` (job → DEAD). On success, job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 0b — Key Data Extraction | None (regex) | Deterministic pre-processor: extracts numbers, percentages, currency amounts, ratios, formulas, underscore identifiers, and date ranges from source text via regex; appends them as a `[Key Data — extracted by pre-processor]` section before any LLM sees the text. Zero token cost; cannot hallucinate. Anchors the synthesis LLM to all quantitative candidates. |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads the full source text (bounded by `max_source_chars`) + BM25 candidates + `purpose.md` scope + status of each candidate page. Outputs per-page action: `create`, `update`, `flag`, `skip`. **RULE 1b:** a page with `status='active'` is human-reviewed and authoritative — when the source provides a conflicting value, date, formula, or conclusion, the action must be `flag`, not `update`. The only exception is a source that adds a topic the page does not yet mention at all. Decision cache version: `v2`. |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. If no citation markers are present in the returned body, a `citation_pass4_no_markers` WARNING audit event is written — the page is still saved but the condition is flagged for re-ingest with a more capable model. Zero-citation results are not cached so a subsequent re-ingest can succeed. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic. **Suppressed when TF fallback was used** (a small corpus with broad coverage may legitimately return fewer than 3 candidates — that is expected, not a gap).\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n| Citation presence (Check 5b) | Page body has ≥ 50 words and zero `^[filename:L-L]` citation markers → WARNING. Diagnostic only — does not block promotion to `active`. Indicates the annotation pass failed to produce markers, usually a model-compatibility issue (use Gemini 2.5 Flash or higher for reliable citation annotation). |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n### ActionAgent\n\nDispatches action-intent queries from the chat UI (e.g. \"activate a draft page\", \"show lint report\", \"archive a stale page\", \"run scaffold\"). Parses the user's free-text intent into a structured action via an LLM extraction prompt, then executes the action against the wiki and returns a human-readable result. When the intent is ambiguous (e.g. \"activate a page\" without specifying which), sets `needs_clarification=True` and returns a prompt and candidate list for the web UI to render as chip buttons.\n\n### ScaffoldAgent\n\nGenerates and updates the wiki's structural pages. Reads all current wiki pages and asks the LLM to organise them into 5–8 domain-appropriate categories, then writes `wiki/index.md` (category index with wikilinks), `wiki/purpose.md` (scope statement), and `AGENTS.md` (ingest and query guidelines). Also stamps a `categories:` field on each page's YAML frontmatter. When `ROUTING.md` already exists, regenerates it to stay in sync with the updated index structure.\n\n### RewriteAgent\n\nRewrites follow-up questions in multi-turn conversations into self-contained, standalone form before BM25 retrieval. Converts context-dependent phrases (\"What came after that?\", \"Tell me more about his early life\") into explicit questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the correct pages without relying on conversation context. Only invoked when conversation history is non-empty. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### SummarizeAgent\n\nCompresses the oldest conversation turns into a single `[Session summary: …]` assistant turn when a session exceeds `conversation_history_turns`. Prevents unbounded context growth across long sessions. Emits a `notice` SSE event the first time compression occurs so the user can see that earlier context was condensed. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### ContextAgent\n\nBuilds token-budgeted context packs for MCP tool consumers. See [Section 20 — Context Packs](#20-context-packs) for full detail.\n\n### SearchDecomposeAgent\n\nDecomposes a web search intent into 1–4 terse keyword search strings optimised for authoritative source retrieval. Uses a separate prompt from `QueryAgent`'s question decomposition — search decomposition asks \"what distinct search strings would find the best sources?\" (terse keyword phrases) rather than \"what distinct questions does this ask?\" (natural-language sub-questions). See [Web search fan-out](#web-search-fan-out) in the IngestAgent section.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n| `session` | `.jsonl` | `claude session`, `codex session`, `cursor session`, `ai session`, `session history` | Extracts human-readable turns from AI coding session transcripts. Supports Claude Code JSONL format and Codex/Cursor format. Filters tool calls, thinking blocks, and sub-agent scaffolding. Short turns (< 20 assistant words, < 3 user words) are skipped. No external dependencies. |\n\n### Session Skill — AI Session History Ingestion\n\nThe `session` skill turns AI coding session history files (`.jsonl`) into searchable wiki pages capturing the problem-solving knowledge exchanged during the session.\n\n**Supported formats**\n\n| Format | File origin | Detection |\n|--------|-------------|-----------|\n| Claude Code | `~/.claude/projects//.jsonl` | `obj.type` ∈ `{\"user\", \"assistant\"}` with nested `obj.message` |\n| Codex / Cursor | Export from OpenAI Codex or Cursor IDE | `obj.role` at top level, no `message` wrapper |\n\nFormat is auto-detected from the first 30 parseable lines. Files that match neither format are parsed as Codex (fallback).\n\n**Filtering rules**\n\n| Content | Kept? | Reason |\n|---------|-------|--------|\n| User text ≥ 3 words | ✓ | Substantive question or instruction |\n| Assistant text ≥ 20 words | ✓ | Substantive answer |\n| User text < 3 words (`\"ok\"`, `\"yes\"`) | ✗ | Too terse to be useful |\n| Assistant text < 20 words | ✗ | Acknowledgement or one-liner |\n| `isSidechain: true` lines | ✗ | Sub-agent scaffolding (not the user's conversation) |\n| `tool_use` blocks | ✗ | Shell commands / file writes — not final output |\n| `tool_result` blocks | ✗ | Raw output — avoids leaking file contents or credentials |\n| `thinking` blocks | ✗ | Internal reasoning — not the final answer |\n| `permission-mode`, `system`, `last-prompt` lines | ✗ | Session metadata, not conversation |\n\nAfter extraction the text passes through Synthadoc's standard pre-LLM source sanitizer\n(zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars,\ninstruction-override phrases) — the same step applied to every PDF, DOCX, URL, and other source type.\nSee [§29 Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer).\n\n**Output format**\n\nEach kept turn is labelled and separated by `---`:\n\n```\n[USER]\nHow do I implement a sliding window algorithm in Python?\n\n---\n\n[ASSISTANT]\nA sliding window algorithm maintains a contiguous subarray by advancing two\npointers simultaneously…\n```\n\n**`suggested_slug`**\n\nThe skill sets `metadata[\"suggested_slug\"]` to `session-YYYY-MM-DD-` where the date comes from the file's `mtime` and the topic from the first substantive user turn (first 6 words, slugified). Example: `session-2026-07-15-how-do-i-implement-sliding`.\n\n**Limitations**\n\n- No chunking in v1.1 — very long sessions are truncated at `max_source_chars` (default 400 000 chars). Split large archives into individual `.jsonl` files.\n- Tool output is excluded by design — re-ingest the original source files if you need the file contents in the wiki.\n- Re-ingesting the same `.jsonl` file uses Synthadoc's standard source-hash dedup — no duplicate pages are created.\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n**Compound identifier expansion:** When a token contains underscores, the tokenizer emits both the compound form and each component as separate tokens (`capex_growth` → `[\"capex_growth\", \"capex\", \"growth\"]`). This expansion is applied at both index time and query time so that human-friendly queries match programmatic identifiers and vice versa. Leading/trailing underscores are stripped before splitting; no empty strings are emitted.\n\n**TF fallback for small corpora:** When all BM25 scores in a result set are zero or negative (IDF collapse on a corpus of ≤ 2 documents, or a term present in every page), the search system falls back to normalised term frequency (TF) scoring: `TF(doc, query) = Σcount(term, doc) / len(doc_tokens)`. Results are returned in the same format as BM25 results with a `tf_fallback: True` flag on each `SearchResult`. This field is recorded in the audit trail and consumed by gap detection (see below).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp, cascade_links_removed_from: [str]}` _(cascade field added v1.0.2)_ |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n| `GET` | `/graph` _(v1.0.0)_ | — | `{status, node_count, edge_count, cluster_count, nodes: [...], edges: [...]}` or `{status: \"computing\"}` on first call |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n**External file paths** (outside the wiki root) are supported when the request comes from\n`127.0.0.1` or `::1` and the payload includes `\"allow_external_paths\": true`. The CLI sets\nthis flag automatically for all local file sources. Remote clients cannot set this flag\n(the server ignores it for non-localhost connections) to prevent arbitrary file reads\nin server-exposed deployments.\n\nThis is required for ingesting files that live outside the wiki directory by design —\nfor example, Claude Code session transcripts at `~/.claude/projects//.jsonl`.\n\n**Client behaviour summary:**\n\n| Client | Sends `allow_external_paths` | Can ingest outside wiki root? |\n|--------|------------------------------|-------------------------------|\n| CLI (`synthadoc ingest `) | `true` (automatic, local file paths only) | Yes — server is localhost |\n| Obsidian plugin | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Web UI | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Direct HTTP API (localhost) | Set manually in request body | Yes — if request from 127.0.0.1/::1 |\n| Direct HTTP API (remote) | Ignored by server | No — server silently treats as `false` |\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 1.1.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n**Reading View default:** `synthadoc plugin install` and `synthadoc plugin upgrade` write `\"defaultViewMode\": \"preview\"` to `.obsidian/app.json` after copying the plugin files. This causes Obsidian to open new notes in Reading View by default, which is required for `^[filename:L-L]` citation chips to render. Only `defaultViewMode` is written; all other keys in `app.json` are preserved. If `app.json` is malformed JSON the write is skipped and a warning is logged — the install still succeeds.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports two things: dangling slugs (pages listed in ROUTING.md that no longer exist in the wiki) and unassigned slugs (pages that exist in index.md but are not listed in any ROUTING.md branch). Enabled only when ROUTING.md exists. **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n| `Graph: show knowledge graph` _(v1.1.0)_ | Open the knowledge graph panel — draggable modal with Canvas force-directed graph, type filter dropdown, cluster legend, hover tooltip showing title/slug/type/state/cluster/connections, and click-to-open page in current pane. System pages (index, overview, dashboard, purpose, log) are excluded. Capped at 300 most-connected nodes for large wikis with an explanatory banner. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [-w wiki] — generate ROUTING.md from index.md branch structure\n│ ├── validate [-w wiki] — report dangling slugs and cross-branch duplicates\n│ └── clean [-w wiki] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [-w wiki]\n├── candidates\n│ ├── list [-w wiki]\n│ ├── promote |--all [-w wiki]\n│ └── discard |--all [-w wiki]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** subscription, or want to use the free **Opencode Zen** tier, you can use either as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode Zen (free, no API key — requires connecting first: run `opencode` → `/connect` → select Zen):\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"opencode/big-pickle\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 30\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n# Citation pass (Pass 4) tuning — these two settings work together:\n# citation_source_lines — how many lines of the source the LLM sees when placing ^[...] markers.\n# Increase if lint reports out_of_range on long sources (transcripts, PDFs).\n# citation_max_tokens — output token budget for the annotated section returned by the LLM.\n# Increase if you raise citation_source_lines and have long wiki sections.\n# citation_source_lines = 400\n# citation_max_tokens = 8192\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax`, `qwen` (DashScope), and `deepseek` providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `30` | Exponential backoff base; delays are `min(base × 2^(attempt−1), 300)` seconds |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Emit `logger.warning` in server log when per-job ingest cost exceeds this threshold; job continues normally |\n| `cost.hard_gate_usd` | float | `2.00` | Permanently fail the ingest job (DEAD) and record a `cost_gate_exceeded` audit event when per-job cost exceeds this threshold |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × (backup_count + 1)` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n| `ingest.max_source_chars` | int | `32000` | Character limit applied to each source before the LLM call. Sources exceeding this limit are truncated; the page's `sources:` frontmatter entry gets `truncated: true` and lint emits a warning. Override per-run with `--max-source-chars N`. _(v1.0.0)_ |\n| `ingest.citation_source_lines` | int | `400` | Number of source lines the LLM sees during Pass 4 (citation annotation). Increase if lint reports `out_of_range` on long sources such as transcripts or PDFs. Raise `citation_max_tokens` in proportion when increasing this value. |\n| `ingest.citation_max_tokens` | int | `8192` | Output token budget for the annotated section returned by Pass 4. Increase when `citation_source_lines` is raised or wiki sections are long, to avoid truncated annotations. |\n| `query.context_wiki_pct` | float | `0.60` | Fraction of the model context window reserved for wiki source pages. _(v1.0.0)_ |\n| `query.context_history_pct` | float | `0.20` | Fraction reserved for conversation history. _(v1.0.0)_ |\n| `query.context_system_pct` | float | `0.15` | Fraction reserved for system prompt and instructions. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_index_pct` | float | `0.05` | Fraction reserved for the wiki index preamble. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_window` | int | _(auto)_ | Override the built-in context window lookup for the configured model. Use when running a local or custom model whose window size is not in the built-in table. _(v1.0.0)_ |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement is wired into the ingest pipeline: `Orchestrator._guard_ingest_cost()` is called after each ingest LLM call with the returned cost, before the job is marked complete.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | `logger.warning` in server log; job continues normally |\n| `hard_gate_usd` | $2.00 | Server/ingest path: records `cost_gate_exceeded` audit event + permanently fails job (DEAD). Interactive CLI: prompts `Proceed? [y/N]` |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `min(backoff_base_seconds × 2^(attempt − 1), 300)` \nWith the default base of 30 s: attempt 1 waits 30 s, attempt 2 waits 60 s, attempt 3 waits 120 s (all capped at 300 s). The `retry_after` timestamp is stored in the jobs table; `dequeue()` skips any job whose `retry_after` is still in the future. Applied only to transient errors (network timeouts, connection failures, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` by default. The bind address is configurable via `server.host` in `config.toml` (e.g. `host = \"0.0.0.0\"` to expose on all interfaces for LAN access). No built-in authentication — restrict via firewall when exposing beyond loopback. Remote access without a reverse proxy is not recommended on shared or networked machines.\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs and unassigned slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `-w ` / `--wiki ` to target a specific wiki.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; when all local source files are missing from disk the page is transitioned to `archived` (if only some sources are missing it is transitioned to `stale` instead). Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: lint evaluates all local (non-URL) source paths listed in the page's frontmatter. If **all** of them are missing from disk, the page is transitioned to `archived`. If only **some** are missing — meaning valid content from at least one source still exists — the page is transitioned to `stale` instead, preserving it for re-ingest rather than discarding it. _(v1.0.2: previously any single missing source triggered an immediate archive.)_\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Cascade link cleanup\n\nWhenever a page transitions to `archived` — whether triggered manually (`synthadoc lifecycle archive`), via the Obsidian Lifecycle modal, the MCP `synthadoc_lifecycle` tool, or automatically by lint detecting a missing source — Synthadoc immediately scans all other active pages and removes every `[[archived-slug]]` wikilink pointing to the now-archived page.\n\n- **Inline links** — `[[archived-slug]]` becomes plain text (the display text if an alias was used, otherwise the slug itself).\n- **List-item links** — a list item whose only content is `[[archived-slug]]` is dropped entirely.\n- **Archived pages are skipped** — pages already archived are not rewritten.\n- **System pages are skipped** — index, overview, dashboard, log, and purpose pages are excluded.\n\nThe cleanup runs synchronously before the response is returned, so the wiki is consistent immediately. The CLI and MCP responses include a `cascade_links_removed_from` field listing the affected slugs. During `lint run`, cascade is batched once after all auto-archive transitions complete, and the count is added to `dangling_links_removed` in the lint report.\n\nLint's existing periodic dangling-link cleanup (`lint run --scope orphans`) remains in place as a safety net for any dead links that predate this feature or arrive through other paths.\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [...], \"input_tokens\": 1240, \"output_tokens\": 380, \"tokens_used\": 1620}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […], \"input_tokens\": N, \"output_tokens\": N, \"tokens_used\": N}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Streaming Token Audit\n\nStreaming LLM responses do not return token counts in the same way as blocking calls. To track real usage in the audit trail:\n\n**Provider-side collection** — each `LLMProvider` subclass exposes two instance attributes: `last_stream_input_tokens` and `last_stream_output_tokens`. These default to `0` and are populated from the API-specific usage field when the stream ends:\n\n| Provider | Mechanism | Verified |\n|---|---|---|\n| OpenAI | `stream_options={\"include_usage\": True}` passed to `chat.completions.create()`; final chunk carries `chunk.usage.prompt_tokens` / `completion_tokens` with empty `choices` | ✅ |\n| Anthropic | `message_start` event → `event.message.usage.input_tokens`; `message_delta` event → `event.usage.output_tokens` | ✅ |\n| Ollama | Final chunk with `done=True` → `prompt_eval_count` / `eval_count` | ✅ |\n| DeepSeek | Same `OpenAIProvider` path as OpenAI; DeepSeek's OpenAI-compatible API supports `stream_options` | ✅ |\n| MiniMax (reasoning: M2.5+) | Detects `` in stream → falls back to blocking `complete()` → captures exact counts from `resp.usage` | ✅ |\n| MiniMax (non-reasoning, e.g. M3 thinking=disabled) | Same `OpenAIProvider` path; MiniMax API silently ignores `stream_options`. Falls back to character-based estimate (÷ 3.5 chars/token) from prompt + answer lengths. Accuracy ±20%. | ✅ estimated |\n| Gemini | Same `OpenAIProvider` path via `generativelanguage.googleapis.com/v1beta/openai/`; Google's compatibility layer honours `stream_options` — verified live (Gemini 2.5 Flash Lite, 50K tokens) | ✅ |\n| Groq | Same `OpenAIProvider` path; Groq's OpenAI-compatible API supports `stream_options` | ✅ |\n| Qwen (DashScope) | Same `OpenAIProvider` path via DashScope; `stream_options` honoured — verified live (qwen-plus, 28K tokens) | ✅ |\n| Qwen (Ollama) | Ollama path → `prompt_eval_count` / `eval_count` | ✅ |\n\n**Forwarding to `done` event** — after `complete_stream()` is exhausted, `QueryAgent.run_stream()` reads the provider attributes and includes them in the final `done` event payload: `input_tokens`, `output_tokens`, `tokens_used`.\n\n**Audit record** — `Orchestrator.query_stream()` consumes the `done` event and calls `estimate_cost(model, input_tokens, output_tokens, is_local)` from `synthadoc/providers/pricing.py`, then calls `AuditDB.record_query()` with the real token count and computed cost. The audit row at `GET /audit/queries` will therefore always show non-zero values for cloud providers.\n\nWhen a provider does not report streaming usage (e.g. a future provider not yet wired), the attributes stay at `0`; the audit row records `tokens=0, cost_usd=0.0` rather than crashing, matching the pre-fix behaviour.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 5) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 5 # turns to include in each request (default: 5; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp, cascade_links_removed_from: [str]}` or `{error, cascade_links_removed_from: []}` _(cascade field added v1.0.2)_ | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions for Codex CLI / OpenCode (if present)\n├── CLAUDE.md ← same content; loaded by Claude Code (if present)\n├── GEMINI.md ← same content; loaded by Gemini CLI (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── hooks/ ← user hook scripts (always included)\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ ├── extracted/ ← text sidecars + PDF pagemaps (always included)\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `logs/`.\n\n> **v0.9.3 fix:** `hooks/` and `.synthadoc/extracted/` were previously missing from backups. Omitting `extracted/` caused the Source Viewer and Provenance citation modal to show \"Could not read source file\" after restore.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## 29. Pre-LLM Source Sanitizer\n\nEvery source document passes through a sanitizer immediately after text extraction, before any LLM call. The sanitizer removes six categories of content that could manipulate the LLM or degrade compilation quality:\n\n| Category | Action | Warning logged? |\n|---|---|---|\n| Zero-width characters (U+200B, U+200C, U+200D, U+FEFF) | Removed silently | No |\n| Bidi override characters (U+202A–U+202E, U+2066–U+2069) | Removed | Yes |\n| HTML comments (``) | Removed silently | No |\n| Hidden CSS spans (`display:none`, `visibility:hidden`) | Removed silently | No |\n| Base64 blobs ≥ 200 consecutive characters | Replaced with `[base64 content removed]` | Yes |\n| Instruction-override phrases (8 patterns: \"ignore previous instructions\", \"disregard the above\", \"override your system prompt\", etc.) | Replaced with `[redacted]` | Always |\n\nWhen content is removed and a warning is appropriate, the server logs a `WARN` entry:\n```\n[WARN] sanitizer stripped content from 'papers/survey.pdf': bidi overrides, instruction-override phrase\n```\n\nThis is a pure pre-processing step with zero LLM cost. Truncation (if configured) is applied after sanitization, so the character budget is consumed by clean text only.\n\n---\n\n## 30. Per-Source Truncation Flag\n\nLarge source documents are truncated before they reach the LLM. By default the limit is **32,000 characters** (~8,000 tokens), raised from the prior hardcoded 8,000-character limit. When a source exceeds this limit, its compiled wiki page records a `truncated: true` flag in its `sources:` frontmatter entry.\n\n### Configuration\n\n```toml\n[ingest]\nmax_source_chars = 32000 # default; raise for large PDFs or books\n```\n\n### Per-ingest override\n\nThe limit can be raised for a single ingest run without modifying the config:\n\n```bash\nsynthadoc ingest papers/large-textbook.pdf --max-source-chars 128000\n```\n\nThe same `max_source_chars` field is accepted by the HTTP `POST /jobs/ingest` body and the MCP `synthadoc_ingest` tool.\n\n### Lint surfacing\n\n`synthadoc lint` emits a warning for any page that has a truncated source:\n\n```\n[WARN] quantum-computing.md: source 'papers/large-survey.pdf' was truncated at ingest\n (source exceeded max_source_chars=32000 — 87,412 chars in source).\n To re-ingest with a higher limit (this source only):\n synthadoc ingest papers/large-survey.pdf --max-source-chars 128000\n To raise the limit for all future ingests:\n set [ingest] max_source_chars = 128000 in your config\n```\n\n---\n\n## 31. Proportional Context Budget\n\nQuery context is allocated proportionally to the configured model's context window, replacing the prior hard cap. The default allocation:\n\n| Slice | Default | Purpose |\n|---|---|---|\n| `context_wiki_pct` | 60% | Wiki page content (ranked by BM25 + vector score) |\n| `context_history_pct` | 20% | Chat turn history (newest-first, oldest dropped when over budget) |\n| `context_system_pct` | 15% | System prompt and purpose document |\n| `context_index_pct` | 5% | `ROUTING.md` and search index |\n\nPercentages must sum to ≤ 100. Configure in `[query]`:\n\n```toml\n[query]\ncontext_window = 0 # 0 = auto-detect from known model map\ncontext_wiki_pct = 60\ncontext_history_pct = 20\ncontext_system_pct = 15\ncontext_index_pct = 5\n```\n\n> **Note:** `context_system_pct` and `context_index_pct` are parsed and validated but not yet enforced as hard caps in this release — the system prompt uses a conservative fixed limit, which is well within the configured 15% slice for all supported models. Full per-slice enforcement is planned for v1.1.\n\n### Model context window map\n\nSynthadoc ships a built-in prefix-matched table:\n\n| Model | Context window |\n|---|---|\n| `claude-opus-4*`, `claude-sonnet-4*`, `claude-haiku-4*` | 200,000 tokens |\n| `gpt-4o*`, `gpt-4-turbo*` | 128,000 tokens |\n| `gpt-4` (exact) | 8,192 tokens |\n| `gpt-3.5-turbo*` | 16,385 tokens |\n| Unknown / fallback | 128,000 tokens |\n\nSet `context_window = N` in config to override the map (useful for local Ollama models with non-standard context sizes).\n\nAt the defaults on a 200k-token model: wiki slice ≈ 480,000 chars — far more than any practical query needs. On constrained models, the budget scales down automatically so the same config works across model families.\n\n---\n\n## 32. Knowledge Graph\n\nSynthadoc computes a wikilink graph across all active and draft pages during every lint run. The graph is stored in two tables in `audit.db`:\n\n```sql\nCREATE TABLE graph_nodes (\n slug TEXT PRIMARY KEY,\n cluster_id INTEGER NOT NULL,\n updated_at TEXT NOT NULL\n);\n\nCREATE TABLE graph_edges (\n from_slug TEXT NOT NULL,\n to_slug TEXT NOT NULL,\n weight INTEGER NOT NULL DEFAULT 1,\n edge_type TEXT NOT NULL DEFAULT 'mixed',\n PRIMARY KEY (from_slug, to_slug)\n);\n```\n\nEdge weight is a composite signal from two sources:\n\n| Edge type | Source | Weight contribution |\n|-----------|--------|---------------------|\n| `wikilink` | `[[slug]]` occurrences in page body | +1 per occurrence |\n| `co_source` | pages compiled from the same source file (matched by hash) | +2 per shared source |\n\nMost edges are `mixed` (have both types). Pure `co_source` edges surface hidden relationships — pages compiled from the same source document are linked immediately after ingest, before any wikilinks are created. `edge_type` is stored on each edge so the web UI can render co-source edges with a different visual style.\n\nCluster IDs are assigned by Louvain community detection — pages that link densely to each other end up in the same cluster.\n\n### REST API\n\n**`GET /graph`** — returns the full graph with per-node enrichment:\n\n```json\n{\n \"status\": \"ready\",\n \"node_count\": 45,\n \"edge_count\": 123,\n \"cluster_count\": 4,\n \"nodes\": [\n {\"slug\": \"quantum-error-correction\", \"title\": \"Quantum Error Correction\",\n \"type\": \"concept\", \"state\": \"active\", \"cluster_id\": 2}\n ],\n \"edges\": [\n {\"from\": \"quantum-error-correction\", \"to\": \"shor-algorithm\", \"weight\": 3, \"edge_type\": \"mixed\"}\n ]\n}\n```\n\nWhen the graph tables are empty (first call after upgrade, or before the first lint run), the server returns `{\"status\": \"computing\"}` and starts a background build immediately. The web UI polls every 2 seconds and renders the graph when the status becomes `\"ready\"`.\n\n### Web UI\n\nThe **Graph** tab (alongside Chat in the top nav) renders a D3.js force-directed graph. Nodes are colored by cluster. Edge thickness is proportional to weight (1–4 px). Pure `co_source` edges (no wikilink) are rendered with a dashed stroke, making hidden source-based relationships visually distinct from explicit cross-links. Clicking a node opens a sidebar showing the page title, type and lifecycle state badges, cluster assignment, and an **\"Ask about this →\"** button that switches to the Chat tab with a pre-filled query.\n\nControls: zoom in/out (scroll or pinch), drag nodes, filter by page type.\n\n### System page exclusion _(v1.1.0)_\n\nThe five structural system pages — `overview`, `index`, `dashboard`, `purpose`, and `log` — are excluded from both `graph_nodes` and `graph_edges`. These pages link to nearly every other page and would act as false hubs, inflating edge counts and distorting Louvain cluster detection. System page exclusion is applied inside `_build_graph()` before any edge computation.\n\n### Obsidian knowledge graph panel _(v1.1.0)_\n\n→ Full description: [§33 Multi-Platform Agent Skill Files](#33-multi-platform-agent-skill-files) covers scaffold; the graph panel itself is documented in [§8 Obsidian Plugin](#8-obsidian-plugin) command table.\n\nThe Obsidian plugin includes a **draggable modal** knowledge graph panel (`Graph: show knowledge graph` command). It fetches from the same `GET /graph` endpoint as the web UI and renders a force-directed Canvas 2D simulation using Verlet integration (no external library). System pages are already excluded from the server response.\n\n| Feature | Obsidian panel | Web UI Graph tab |\n|---------|----------------|-----------------|\n| Engine | Canvas 2D + Verlet | D3.js SVG |\n| Node click | Opens the wiki page in the current pane | Opens the node sidebar with \"Ask about this →\" |\n| **Best for** | **Navigating to and editing pages** | **Asking questions about a page** |\n| Draggable panel | Yes — drag header to reposition | Fixed tab layout |\n| Node cap | 300 most-connected (see note) | No cap |\n| Cluster legend | Yes | Yes |\n| Type filter | Yes | Yes |\n\n**Rule of thumb:** Use the **web UI** when you want to ask questions about a node. Use the **Obsidian panel** when you want to navigate to and edit a page.\n\n**Node cap rationale:** The Obsidian panel uses a hand-written Verlet integrator with O(n²) pairwise charge repulsion — every node repels every other node on each animation tick. At 300 nodes that is ~90,000 operations per frame, which runs smoothly at 60 fps inside Obsidian's Electron renderer. At 500 nodes it is ~250,000 operations per frame and starts to lag noticeably. The web UI has no cap because D3's `forceManyBody` uses a Barnes-Hut tree approximation (O(n log n)), which scales to thousands of nodes without frame-rate issues. When the cap is hit, nodes are ranked by degree and the 300 most-connected are shown; a banner reports the total page count and the number of hidden nodes, and suggests using the type filter to bring the set below 300.\n\n→ User walkthrough: [Quick-Start Guide §24 — Knowledge Graph](docs/user-quick-start-guide.md#knowledge-graph)\n\n---\n\n## 33. Multi-Platform Agent Skill Files\n\n\n\n`synthadoc init` and `synthadoc scaffold` write three agent skill files at the wiki root:\n\n| File | Format | Read by |\n|------|--------|---------|\n| `AGENTS.md` | OpenAI Agents SDK / Codex / OpenCode | Codex CLI, OpenCode, generic agents |\n| `CLAUDE.md` | Claude Code / Anthropic | Claude Code (loaded automatically when the wiki folder is opened) |\n| `GEMINI.md` | Gemini CLI | Gemini CLI |\n\nAll three files carry **identical content**: a title block, the LLM-generated domain guidelines for the wiki, a quick-reference CLI command table, and an MCP tool table. The only difference is the heading comment. This means any AI coding tool that opens the wiki root gets full Synthadoc guidance without manual setup.\n\n### Content generated by `scaffold`\n\nEach skill file is generated from the same LLM call that produces `purpose.md` and `index.md`. The LLM receives the current wiki page list and domain description, then generates:\n\n1. **Domain label** — a precise human-readable name inferred from wiki content (e.g. `History of Computing`, not `My Wiki`)\n2. **Domain guidelines** — 3 domain-specific ingest and query guidelines as plain sentences\n3. **Quick Reference table** — pre-filled with the current wiki name and port\n\nIf the wiki was created before v1.1.0, running `synthadoc scaffold` once regenerates all three files and adds `CLAUDE.md` and `GEMINI.md` to any wiki that only had `AGENTS.md`.\n\n### Template source\n\nThe instruction body is defined in `synthadoc/cli/_init.py` as `_AGENT_INSTRUCTION_BODY` and shared by all three templates, keeping the files permanently in sync. The scaffold agent substitutes `{domain}`, `{guidelines}`, and `{port}` at write time.\n\n---\n\n## 34. Synthadoc Scaffold Marker\n\n\n\nThe scaffold marker `` (written as an HTML comment) separates **user-authored content** (preserved) from **scaffold-managed content** (rewritten on each `synthadoc scaffold` run).\n\n### Single-marker mode (index.md)\n\nThe marker appears once, below the H1 heading in `index.md`. Everything above the marker is the user zone; everything below is replaced with new LLM-generated categories.\n\n```markdown\nMy custom introduction — maintained by hand.\n\n\n\n## Pioneers and Visionaries\n- [[alan-turing]]\n...\n```\n\nIf the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### Per-section multi-marker mode (purpose.md)\n\n`purpose.md` uses a **per-section** approach: the marker appears inside each `## Section` heading. Within a section, content the user writes **above** the marker is preserved; content below is replaced with fresh LLM output. Sections without a marker in the existing file are replaced entirely (and a marker is added). Sections in the existing file that the scaffold template does not include are kept unchanged.\n\n```markdown\n## Overview\n\nMy custom overview note.\n\n\n\nLLM-generated overview content goes here.\n\n## What Belongs\n\n\n\nLLM-generated scope definition.\n```\n\nThis design lets users annotate any section of `purpose.md` without losing their notes on the next scaffold run.\n\n### Marker preservation guarantee\n\n- `preserve_user_zone()` in `scaffold_agent.py` handles both modes.\n- If a section has no marker in the existing file, `preserve_user_zone()` treats the entire section body as user content and preserves it, replacing only what falls below any marker it finds.\n- The scaffold NEVER touches `config.toml`, `dashboard.md`, or any wiki page (`wiki/*.md`).\n\n→ User walkthrough: [Quick-Start Guide §14 — Enrich the wiki with scaffold](docs/user-quick-start-guide.md#scaffold)\n\n---\n\n## Customization\n\nSynthadoc exposes four extension points — all hot-loaded, no server restart required.\n\n### Custom skills (new file formats)\n\nSubclass `BaseSkill` (Apache-2.0 — no AGPL obligation on your skill code), drop the folder in `/skills/` or `~/.synthadoc/skills/`, and Synthadoc picks it up on the next ingest. Skills match by file extension or intent prefix and support any Unicode text, including CJK prefixes.\n\n→ Full interface, manifest format, and examples: [§5 Skills System](#5-skills-system) and [§17 Plugin Development Guide](#17-plugin-development-guide)\n\n### Custom LLM providers\n\nSubclass `LLMProvider` from `synthadoc/providers/base.py` (Apache-2.0) and place the file in `~/.synthadoc/providers/` or the wiki `providers/` directory. Switch active provider with one config line.\n\n→ Full interface and wiring instructions: [§10 Configuration — Provider switching](#10-configuration)\n\n### Hooks\n\nShell scripts (any language) that fire on `on_ingest_complete` and `on_lint_complete`. Receive a JSON context on stdin. Set `blocking = true` to gate the operation on the hook's exit code — useful for CI pipelines and quality gates.\n\n→ Full event schema, context JSON, and blocking behaviour: [§11 Hook System](#11-hook-system)\n\n### Cache control\n\nThree cache layers (embedding, LLM response, provider prompt cache) invalidate automatically on source-file change (SHA-256). Force a fresh call with `--force`, or wipe all cached responses with `synthadoc cache clear -w `.\n\n→ Layer-by-layer breakdown and invalidation rules: [§12 Cache System](#12-cache-system)\n\n### Multi-Platform Agent Skill Files\n\nEvery Synthadoc wiki ships three companion files that give AI coding agents a complete, self-contained operating guide for the wiki:\n\n| File | Platform |\n|---|---|\n| `AGENTS.md` | Codex CLI, OpenCode, and any agent that reads the AGENTS.md convention |\n| `CLAUDE.md` | Claude Code (this session's tool) — highest-priority instruction source in Claude's hierarchy |\n| `GEMINI.md` | Gemini CLI — relevant because Synthadoc's default LLM provider is `gemini-2.5-flash-lite` |\n\nAll three files share identical body content generated from the same template; they differ only in their H1 heading so each agent recognises its own file first. The body covers:\n\n- **Domain guidelines** — LLM-generated bullet list of domain-specific ingest and query rules (produced by `scaffold`)\n- **Quick reference table** — every common CLI command with the correct `synthadoc` syntax\n- **Server startup** — how to start the server and verify it is running\n- **Ingest examples** — local file, URL, YouTube, agent session `.jsonl`, force reingest, dry-run\n- **Query** — basic and streaming query syntax; citation marker format\n- **Lint** — when to run and what each check does\n- **Lifecycle** — transition commands with state names\n- **Page schema** — frontmatter structure with all fields\n- **MCP tools table** — all 12 tools with purpose descriptions\n\n**How the files are created and updated:**\n\n1. `synthadoc init` — writes all three files with default domain guidelines and the wiki's configured port.\n2. `synthadoc scaffold` — regenerates all three with LLM-produced domain guidelines derived from the current wiki state. The `scaffold` command prints a confirmation line for each file.\n\n**Use cases:**\n\n- Open Claude Code in a Synthadoc wiki directory and it immediately knows how to ingest, query, lint, and manage lifecycle — no additional setup. Claude Code can also start the server in background mode (`synthadoc serve -w -b`), stop it cleanly, and orchestrate multi-step workflows, effectively acting as a control plane for the wiki.\n- Gemini CLI reads `GEMINI.md` and can run `synthadoc` commands with the correct flags without asking the user for help.\n- CI pipelines with Codex CLI use `AGENTS.md` to drive automated lint and lifecycle transitions on PR merge.\n- Agent session files produced by Claude Code (`.jsonl` under `~/.claude/projects/`) can themselves be ingested into the wiki, capturing the reasoning behind design decisions alongside the source documents.\n\n→ See [README.md — Interfaces & Integration](../README.md#interfaces--integration) for the feature comparison row and quick-start instructions.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v1.1.2\n\n- **Exponential backoff on transient URL fetch failures** — jobs failing with transient errors (connection timeouts, 5xx responses, dropped TCP connections including `RemoteProtocolError`) now re-queue with exponential backoff: `min(base × 2^(attempt−1), 300)` seconds (default base 30 s). Previously all failures retried immediately, exhausting the retry budget within seconds. `dequeue()` skips any job whose `retry_after` timestamp is still in the future; other jobs run normally during the wait window. See [§14 Job Queue](#14-job-queue).\n- **Accurate retry outcome in job warning logs** — the transient-failure warning now reads the current attempt count from the database before logging, so it correctly reports `will retry` vs. `final attempt` (was always `will retry` regardless of retry state).\n- **Scheduler subprocess timeout** — scheduled jobs (`lint`, `ingest --batch`, etc.) are now guarded by `asyncio.wait_for(proc.communicate(), timeout=job_timeout_seconds)`; a hung subprocess is killed after the timeout and the run is recorded as `failed`. Previously a stalled subprocess could block the scheduler task indefinitely. `job_timeout_seconds` defaults to 600 s and is now shown (uncommented) in fresh-init `config.toml`.\n- **Streaming query token and cost accuracy** — the streaming query audit record was written with `tokens=0, cost_usd=0.0`; it now records the actual LLM token usage and cost from the streamed response.\n- **Cost hard gate timing** — the cost hard gate now fires before LLM tokens are spent and pages are written; previously the gate ran after the LLM call completed, allowing partial page writes before rejection.\n- **Session state memory growth bounded** — `_session_state` in the HTTP server is pruned at 100 concurrent sessions (LRU eviction) to prevent unbounded memory growth on long-running servers.\n- **Streamed `` block cleanup** — a leaked fallback asyncio task is now cancelled when a streaming response ends inside an unclosed `` block (e.g. reasoning-model responses cut off mid-think).\n- **Config validation for staging fields** — `staging_policy` values other than `off`, `all`, and `threshold`, and non-numeric `staging_confidence_min`, were silently accepted and caused runtime errors; they now raise a `ConfigError` at server startup.\n- **Job status guard on retry** — retrying a job that is already `in_progress`, `done`, or `skipped` now returns an error instead of silently resetting it to `pending`.\n- **Jobs endpoint performance** — `GET /jobs/{id}` and `DELETE /jobs/{id}` now use a direct primary-key lookup instead of a full table scan; eliminates O(N) reads on large job queues.\n- **Vector migration GC safety** — the background vector migration `asyncio.Task` is now held in a strong reference to prevent premature garbage collection mid-run.\n- **Quota error stack trace suppression** — Qwen 403 errors and Claude Code session-limit exhaustion now emit a concise warning naming the actionable config key, instead of a full stack trace.\n- **Scaffold agent JSON robustness** — `MiniMax-M3` and other reasoning models can truncate mid-JSON; the scaffold agent now handles truncated JSON output gracefully rather than failing the job.\n- **Staging/Candidates section in agent skill files** — `AGENTS.md`, `CLAUDE.md`, and `GEMINI.md` generated by `synthadoc scaffold` now include a `## Staging and Candidates` section.\n- **Citation failure pagination fix** — `GET /audit/citation-failures` was applying the `LIMIT` in SQL before filtering by `page_slug`, returning incorrect results for paginated per-page queries; filtering now runs before the limit.\n\n### v1.1.0\n\n- **Weighted knowledge graph edges** — graph edges now carry two signals: wikilink occurrences (+1 each) and co-source connections — pages compiled from the same source file (matched by SHA-256 hash) — (+2 per shared source). `edge_type` field added to `graph_edges` table (`wikilink`, `co_source`, or `mixed`). `GET /graph` exposes `edge_type` per edge. Web UI renders edge thickness proportional to weight (1–4 px) and dashed lines for pure co-source edges. Co-source edges appear immediately after ingest — before any wikilinks — surfacing hidden relationships in the graph. Schema migrated to version 3.\n- **Multi-platform agent skill files** — `synthadoc init` and `synthadoc scaffold` write `CLAUDE.md`, `AGENTS.md`, and `GEMINI.md` at wiki root; all three carry LLM-generated domain guidelines and the full CLI quick-reference so Claude Code, Codex CLI, OpenCode, Pi, and Gemini CLI users get first-class guidance without manual setup. Run `synthadoc scaffold` once on any pre-v1.1.0 wiki to generate the new files. See [§33](#33-multi-platform-agent-skill-files).\n- **Session history ingestion skill** — `.jsonl` session files from Claude Code and Codex CLI are ingested as wiki pages; human turns and substantive assistant responses extracted; tool calls, thinking blocks, and sub-agent scaffolding skipped; format auto-detected; `suggested_slug` derived from session date and first user message. Converts ephemeral problem-solving knowledge into searchable wiki entries. See [§5 Skills System — Session Skill](#session-skill--ai-session-history-ingestion).\n- **Obsidian knowledge graph panel** — new `Graph: show knowledge graph` command in the Obsidian plugin opens a draggable modal with a Canvas 2D force-directed graph. Features: Verlet integration, type filter dropdown, cluster legend, hover tooltip (title, slug, type, state, cluster, connections), click-to-open page in current pane, node drag to repin. Closes only via the × button — Escape and outside-click do not close the panel, matching the behaviour of other Synthadoc modals. **Large-wiki cap:** when the node set exceeds 300 (after any type filter is applied), nodes are ranked by degree (total edge count) and the 300 most-connected are displayed; edges to capped-out nodes are removed; a banner explains the cap and suggests using the type filter to narrow the view below 300. See [§8 Obsidian Plugin](#8-obsidian-plugin) and [§32 Knowledge Graph — Obsidian panel](#obsidian-knowledge-graph-panel-v110).\n- **Per-section scaffold markers in `purpose.md`** — `purpose.md` now uses per-section `` markers. Users can write content above the marker in any `## Section` block; that content is preserved on every re-scaffold. A single-marker mode for `index.md` continues to work as before. See [§34 Synthadoc Scaffold Marker](#34-synthadoc-scaffold-marker).\n- **System pages excluded from knowledge graph** — `overview`, `index`, `dashboard`, `purpose`, and `log` are filtered out of `_build_graph()` before any edge computation so they cannot act as false hubs in Louvain cluster detection. Both the Obsidian graph panel and the web UI Graph tab reflect this exclusion.\n- **Web UI graph cluster legend** — the web UI Graph tab now shows a cluster legend below the canvas (matching the Obsidian plugin panel), with a color swatch and label for each Louvain cluster present in the filtered node set.\n- **`_merge_section_into_page` deduplication** — force re-ingest of the same source file (e.g. an updated `.jsonl` session) now replaces the page body rather than appending a duplicate section. `_files_match()` normalizes absolute vs. relative paths so the same file is recognized regardless of how it was passed to the ingest command.\n- **`purpose.md` domain label inference** — `scaffold` now infers a precise `domain_label` from current wiki content (e.g. `History of Computing`) instead of using a generic placeholder; the label is used in all three agent skill files and in `purpose.md`'s Overview section.\n- **CI: Windows runner fix** — `scripts/sync_plugin.py` and `scripts/sync_web_ui.py` now use `shutil.copy2` instead of `cp -f`, resolving CI failures on Windows runners for the sync check.\n\n### v1.0.2\n\n- **Cascade link cleanup on archive** — archiving any page immediately removes every `[[slug]]` wikilink pointing to it from all other active pages, without waiting for the next `lint run`. Inline links become plain text; list-item-only links are dropped entirely. The HTTP `POST /lifecycle/transition` and MCP `synthadoc_lifecycle` responses now include a `cascade_links_removed_from` field listing the rewritten page slugs. During `lint run`, cascade runs once after all auto-archive transitions complete and the count is included in the lint report's `dangling_links_removed` field. System pages (index, dashboard, purpose, log, overview) and already-archived pages are skipped. The existing dangling-link lint pass remains as a safety net for dead links that predate this release.\n- **Multi-source partial-archive fix** — when a page has multiple local source files and only some are missing from disk, lint now transitions the page to `stale` (not `archived`), preserving it for re-ingest. The page is only archived when every local source file is gone. URL sources are not affected by this check.\n\n### v1.0.1\n\n- **Scaffold JSON extraction hardened** — replaced the greedy regex used to extract the LLM's JSON response with a brace-balanced parser; eliminates false-positive matches on `[[wikilink]]` syntax that previously caused `json.loads()` failures and silent scaffold corruption\n- **Generated file validation** — scaffold now validates every LLM-generated file (index, purpose, AGENTS.md) for minimum length, required frontmatter, and `[[wikilink]]` presence before writing to disk; validation failure marks the job as failed with a structured error message rather than silently writing a malformed page\n- **ROUTING.md regeneration validation** — `routing init` and scheduled routing regeneration now validate the regenerated file for minimum line count and anchor presence before applying it; prevents a corrupt or truncated ROUTING.md from entering the wiki\n- **Scaffold failure reporting** — scaffold skip and failure reasons are now surfaced in the terminal output and job error field; previously silent failures (e.g. LLM timeout, JSON parse error) are now visible without tailing the JSON log\n- **Install-time scaffold removed** — `synthadoc install` no longer runs the LLM scaffold call during installation; scaffold is a separate step (`synthadoc scaffold`) that the user runs after setting their API key, avoiding confusing API-key-not-set errors at install time\n- **AquaFlow LLM evaluation** — 15-question M&A due-diligence benchmark across five models (MiniMax-Think M3, Claude Opus 4.8, Claude Sonnet 4.6, DeepSeek-R1, Qwen Plus); full report at `docs/example/aquaflow/evaluation/report/llm-query-benchmark.md`\n\n### v1.0.0\n\n- **Pre-LLM source sanitizer** — every source document passes through a sanitizer before any LLM call; removes six categories of potentially harmful content (zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars, instruction-override phrases); a `WARN` log entry is emitted when bidi overrides or instruction-override phrases are stripped; zero LLM cost; applied before truncation so the character budget is consumed by clean text only\n- **Per-source truncation flag** — default character limit raised from 8,000 to 32,000 characters per source (~8,000 tokens); truncated sources are flagged with `truncated: true` in the page's `sources:` frontmatter; `--max-source-chars N` CLI flag and matching `POST /jobs/ingest` body field and MCP `synthadoc_ingest` parameter override the limit per run; `synthadoc lint` emits a warning for any page with a truncated source and suggests the override command; configurable via `[ingest] max_source_chars` in `config.toml`\n- **Proportional context budget** — query context is allocated proportionally to the configured model's context window; four configurable slices: `context_wiki_pct` (60%), `context_history_pct` (20%), `context_system_pct` (15%), `context_index_pct` (5%); built-in prefix-matched model context window table covers Claude 4, GPT-4o/turbo/3.5, and GPT-4; unknown models fall back to 128,000 tokens; `context_window = N` in `[query]` overrides the table for local or custom models\n- **Knowledge graph** — wikilink graph computed across all active and draft pages during every lint run; stored in `graph_nodes` and `graph_edges` tables in `audit.db`; edge weight = number of `[[wikilink]]` references between two pages; cluster IDs assigned by Louvain community detection; `GET /graph` REST endpoint returns nodes with per-node enrichment (title, type, state, cluster) and edges; first call after upgrade triggers a background build and returns `{\"status\":\"computing\"}`; web UI **Graph** tab (D3.js force-directed graph) — nodes colored by cluster, click to see page details and an \"Ask about this →\" button that opens a pre-filled chat query\n- **Citation accuracy & rendering improvements** — six correctness fixes to the ingest citation pipeline: (1) full source text passed to the decision LLM so citations can reference any line in the document; (2) `_annotate_citations` bugs A–F resolved (false-positive matches, off-by-one line ranges, duplicate markers, escaped bracket handling, multi-paragraph span errors, and source filename normalisation); (3) `_has_citations` check added — lint warns and suggests a model upgrade when a page has zero citation markers after synthesis; (4) `Key Data` section extraction hardened so numerical facts, rates, and formulas are preserved verbatim across the annotation pass; (5) `obsidianCitationsToGfm` rendering fixed in the web UI — citation superscripts now display correctly in the chat panel alongside streamed answers; (6) reading view set as the default Obsidian display mode for wiki pages\n\n### v0.9.3\n\n- **Backup now includes `hooks/` and `.synthadoc/extracted/`** — both directories were silently omitted from backups. `hooks/` contains user-customised hook scripts; `.synthadoc/extracted/` contains per-source text sidecars and PDF pagemaps written by the ingest agent. Without these, the Source Viewer and Provenance citation modal showed \"Could not read source file\" after restore.\n- **PyPI publish workflow** — `publish.yml` triggers on GitHub Release, builds the wheel with hatchling, and uploads to PyPI via OIDC trusted publishing (no API token stored). `pip install synthadoc` now works.\n- **Web UI bundled in pip package** — `synthadoc/data/web-ui/dist/` is now included in the wheel. `synthadoc web` previously returned 503 on pip-installed instances because the React dist was not packaged.\n- **CI sync checks** — PRs now fail if `synthadoc/data/obsidian-plugin/`, `synthadoc/data/web-ui/`, or `README-pypi.md` are out of sync with their source. All sync-check steps use `shell: bash` for Windows runner compatibility.\n- **`bump_version.py` extended** — now also updates `synthadoc/data/obsidian-plugin/manifest.json` alongside the source manifest, preventing version drift between the bundled copy and the plugin source.\n\n### v0.9.2\n\n- **`synthadoc/data/web-ui/dist/` packaged** — web UI dist files committed to the repo and bundled in the wheel via hatchling artifacts.\n- **`scripts/sync_web_ui.py`** — helper script to sync `web-ui/dist/` into `synthadoc/data/web-ui/dist/` after `npm run build`.\n\n### v0.9.1\n\n- **PyPI metadata** — added `readme`, `keywords`, `classifiers`, and `[project.urls]` to `pyproject.toml` for correct PyPI rendering.\n- **`README-pypi.md`** — generated from `README.md` by `scripts/prepare_pypi_readme.py`; strips `pypi-strip` blocks and rewrites relative links to absolute GitHub URLs.\n- **Branch protection compatibility** — removed post-merge `build-plugin` CI job that pushed compiled plugin directly to `main`. Plugin sync is now enforced in PRs: CI fails if `synthadoc/data/obsidian-plugin/` is out of sync.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 5). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n"}, "6c8112910993a9098e31e0b1ca674a45cbc71b94": {"t": 1785767246.4437313, "n": 223563, "h": "040a78138269d619", "c": "# Synthadoc — Design Document\n\n**Version:** 1.1.2 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n29. [Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer)\n30. [Per-Source Truncation Flag](#30-per-source-truncation-flag)\n31. [Proportional Context Budget](#31-proportional-context-budget)\n32. [Knowledge Graph](#32-knowledge-graph)\n33. [Multi-Platform Agent Skill Files](#33-multi-platform-agent-skill-files)\n34. [Synthadoc Scaffold Marker](#34-synthadoc-scaffold-marker)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for Codex CLI, OpenCode, and generic OpenAI agents\n CLAUDE.md ← same content as AGENTS.md; loaded automatically by Claude Code\n GEMINI.md ← same content as AGENTS.md; loaded automatically by Gemini CLI\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the full source text (bounded by `max_source_chars`) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `flag` / `skip`). Pages with `status='active'` are protected — sources that conflict with them trigger `flag` rather than `update` (RULE 1b).\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Orchestrator calls `_guard_ingest_cost()` on the returned cost: soft warn → `logger.warning` only; hard gate → `cost_gate_exceeded` audit event + `fail_permanent` (job → DEAD). On success, job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 0b — Key Data Extraction | None (regex) | Deterministic pre-processor: extracts numbers, percentages, currency amounts, ratios, formulas, underscore identifiers, and date ranges from source text via regex; appends them as a `[Key Data — extracted by pre-processor]` section before any LLM sees the text. Zero token cost; cannot hallucinate. Anchors the synthesis LLM to all quantitative candidates. |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads the full source text (bounded by `max_source_chars`) + BM25 candidates + `purpose.md` scope + status of each candidate page. Outputs per-page action: `create`, `update`, `flag`, `skip`. **RULE 1b:** a page with `status='active'` is human-reviewed and authoritative — when the source provides a conflicting value, date, formula, or conclusion, the action must be `flag`, not `update`. The only exception is a source that adds a topic the page does not yet mention at all. Decision cache version: `v2`. |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. If no citation markers are present in the returned body, a `citation_pass4_no_markers` WARNING audit event is written — the page is still saved but the condition is flagged for re-ingest with a more capable model. Zero-citation results are not cached so a subsequent re-ingest can succeed. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic. **Suppressed when TF fallback was used** (a small corpus with broad coverage may legitimately return fewer than 3 candidates — that is expected, not a gap).\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n| Citation presence (Check 5b) | Page body has ≥ 50 words and zero `^[filename:L-L]` citation markers → WARNING. Diagnostic only — does not block promotion to `active`. Indicates the annotation pass failed to produce markers, usually a model-compatibility issue (use Gemini 2.5 Flash or higher for reliable citation annotation). |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n### ActionAgent\n\nDispatches action-intent queries from the chat UI (e.g. \"activate a draft page\", \"show lint report\", \"archive a stale page\", \"run scaffold\"). Parses the user's free-text intent into a structured action via an LLM extraction prompt, then executes the action against the wiki and returns a human-readable result. When the intent is ambiguous (e.g. \"activate a page\" without specifying which), sets `needs_clarification=True` and returns a prompt and candidate list for the web UI to render as chip buttons.\n\n### ScaffoldAgent\n\nGenerates and updates the wiki's structural pages. Reads all current wiki pages and asks the LLM to organise them into 5–8 domain-appropriate categories, then writes `wiki/index.md` (category index with wikilinks), `wiki/purpose.md` (scope statement), and `AGENTS.md` (ingest and query guidelines). Also stamps a `categories:` field on each page's YAML frontmatter. When `ROUTING.md` already exists, regenerates it to stay in sync with the updated index structure.\n\n### RewriteAgent\n\nRewrites follow-up questions in multi-turn conversations into self-contained, standalone form before BM25 retrieval. Converts context-dependent phrases (\"What came after that?\", \"Tell me more about his early life\") into explicit questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the correct pages without relying on conversation context. Only invoked when conversation history is non-empty. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### SummarizeAgent\n\nCompresses the oldest conversation turns into a single `[Session summary: …]` assistant turn when a session exceeds `conversation_history_turns`. Prevents unbounded context growth across long sessions. Emits a `notice` SSE event the first time compression occurs so the user can see that earlier context was condensed. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### ContextAgent\n\nBuilds token-budgeted context packs for MCP tool consumers. See [Section 20 — Context Packs](#20-context-packs) for full detail.\n\n### SearchDecomposeAgent\n\nDecomposes a web search intent into 1–4 terse keyword search strings optimised for authoritative source retrieval. Uses a separate prompt from `QueryAgent`'s question decomposition — search decomposition asks \"what distinct search strings would find the best sources?\" (terse keyword phrases) rather than \"what distinct questions does this ask?\" (natural-language sub-questions). See [Web search fan-out](#web-search-fan-out) in the IngestAgent section.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n| `session` | `.jsonl` | `claude session`, `codex session`, `cursor session`, `ai session`, `session history` | Extracts human-readable turns from AI coding session transcripts. Supports Claude Code JSONL format and Codex/Cursor format. Filters tool calls, thinking blocks, and sub-agent scaffolding. Short turns (< 20 assistant words, < 3 user words) are skipped. No external dependencies. |\n\n### Session Skill — AI Session History Ingestion\n\nThe `session` skill turns AI coding session history files (`.jsonl`) into searchable wiki pages capturing the problem-solving knowledge exchanged during the session.\n\n**Supported formats**\n\n| Format | File origin | Detection |\n|--------|-------------|-----------|\n| Claude Code | `~/.claude/projects//.jsonl` | `obj.type` ∈ `{\"user\", \"assistant\"}` with nested `obj.message` |\n| Codex / Cursor | Export from OpenAI Codex or Cursor IDE | `obj.role` at top level, no `message` wrapper |\n\nFormat is auto-detected from the first 30 parseable lines. Files that match neither format are parsed as Codex (fallback).\n\n**Filtering rules**\n\n| Content | Kept? | Reason |\n|---------|-------|--------|\n| User text ≥ 3 words | ✓ | Substantive question or instruction |\n| Assistant text ≥ 20 words | ✓ | Substantive answer |\n| User text < 3 words (`\"ok\"`, `\"yes\"`) | ✗ | Too terse to be useful |\n| Assistant text < 20 words | ✗ | Acknowledgement or one-liner |\n| `isSidechain: true` lines | ✗ | Sub-agent scaffolding (not the user's conversation) |\n| `tool_use` blocks | ✗ | Shell commands / file writes — not final output |\n| `tool_result` blocks | ✗ | Raw output — avoids leaking file contents or credentials |\n| `thinking` blocks | ✗ | Internal reasoning — not the final answer |\n| `permission-mode`, `system`, `last-prompt` lines | ✗ | Session metadata, not conversation |\n\nAfter extraction the text passes through Synthadoc's standard pre-LLM source sanitizer\n(zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars,\ninstruction-override phrases) — the same step applied to every PDF, DOCX, URL, and other source type.\nSee [§29 Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer).\n\n**Output format**\n\nEach kept turn is labelled and separated by `---`:\n\n```\n[USER]\nHow do I implement a sliding window algorithm in Python?\n\n---\n\n[ASSISTANT]\nA sliding window algorithm maintains a contiguous subarray by advancing two\npointers simultaneously…\n```\n\n**`suggested_slug`**\n\nThe skill sets `metadata[\"suggested_slug\"]` to `session-YYYY-MM-DD-` where the date comes from the file's `mtime` and the topic from the first substantive user turn (first 6 words, slugified). Example: `session-2026-07-15-how-do-i-implement-sliding`.\n\n**Limitations**\n\n- No chunking in v1.1 — very long sessions are truncated at `max_source_chars` (default 400 000 chars). Split large archives into individual `.jsonl` files.\n- Tool output is excluded by design — re-ingest the original source files if you need the file contents in the wiki.\n- Re-ingesting the same `.jsonl` file uses Synthadoc's standard source-hash dedup — no duplicate pages are created.\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n**Compound identifier expansion:** When a token contains underscores, the tokenizer emits both the compound form and each component as separate tokens (`capex_growth` → `[\"capex_growth\", \"capex\", \"growth\"]`). This expansion is applied at both index time and query time so that human-friendly queries match programmatic identifiers and vice versa. Leading/trailing underscores are stripped before splitting; no empty strings are emitted.\n\n**TF fallback for small corpora:** When all BM25 scores in a result set are zero or negative (IDF collapse on a corpus of ≤ 2 documents, or a term present in every page), the search system falls back to normalised term frequency (TF) scoring: `TF(doc, query) = Σcount(term, doc) / len(doc_tokens)`. Results are returned in the same format as BM25 results with a `tf_fallback: True` flag on each `SearchResult`. This field is recorded in the audit trail and consumed by gap detection (see below).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp, cascade_links_removed_from: [str]}` _(cascade field added v1.0.2)_ |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n| `GET` | `/graph` _(v1.0.0)_ | — | `{status, node_count, edge_count, cluster_count, nodes: [...], edges: [...]}` or `{status: \"computing\"}` on first call |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n**External file paths** (outside the wiki root) are supported when the request comes from\n`127.0.0.1` or `::1` and the payload includes `\"allow_external_paths\": true`. The CLI sets\nthis flag automatically for all local file sources. Remote clients cannot set this flag\n(the server ignores it for non-localhost connections) to prevent arbitrary file reads\nin server-exposed deployments.\n\nThis is required for ingesting files that live outside the wiki directory by design —\nfor example, Claude Code session transcripts at `~/.claude/projects//.jsonl`.\n\n**Client behaviour summary:**\n\n| Client | Sends `allow_external_paths` | Can ingest outside wiki root? |\n|--------|------------------------------|-------------------------------|\n| CLI (`synthadoc ingest `) | `true` (automatic, local file paths only) | Yes — server is localhost |\n| Obsidian plugin | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Web UI | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Direct HTTP API (localhost) | Set manually in request body | Yes — if request from 127.0.0.1/::1 |\n| Direct HTTP API (remote) | Ignored by server | No — server silently treats as `false` |\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 1.1.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n**Reading View default:** `synthadoc plugin install` and `synthadoc plugin upgrade` write `\"defaultViewMode\": \"preview\"` to `.obsidian/app.json` after copying the plugin files. This causes Obsidian to open new notes in Reading View by default, which is required for `^[filename:L-L]` citation chips to render. Only `defaultViewMode` is written; all other keys in `app.json` are preserved. If `app.json` is malformed JSON the write is skipped and a warning is logged — the install still succeeds.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports two things: dangling slugs (pages listed in ROUTING.md that no longer exist in the wiki) and unassigned slugs (pages that exist in index.md but are not listed in any ROUTING.md branch). Enabled only when ROUTING.md exists. **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n| `Graph: show knowledge graph` _(v1.1.0)_ | Open the knowledge graph panel — draggable modal with Canvas force-directed graph, type filter dropdown, cluster legend, hover tooltip showing title/slug/type/state/cluster/connections, and click-to-open page in current pane. System pages (index, overview, dashboard, purpose, log) are excluded. Capped at 300 most-connected nodes for large wikis with an explanatory banner. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [-w wiki] — generate ROUTING.md from index.md branch structure\n│ ├── validate [-w wiki] — report dangling slugs and cross-branch duplicates\n│ └── clean [-w wiki] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [-w wiki]\n├── candidates\n│ ├── list [-w wiki]\n│ ├── promote |--all [-w wiki]\n│ └── discard |--all [-w wiki]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** subscription, or want to use the free **Opencode Zen** tier, you can use either as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode Zen (free, no API key — requires connecting first: run `opencode` → `/connect` → select Zen):\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"opencode/big-pickle\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 30\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n# Citation pass (Pass 4) tuning — these two settings work together:\n# citation_source_lines — how many lines of the source the LLM sees when placing ^[...] markers.\n# Increase if lint reports out_of_range on long sources (transcripts, PDFs).\n# citation_max_tokens — output token budget for the annotated section returned by the LLM.\n# Increase if you raise citation_source_lines and have long wiki sections.\n# citation_source_lines = 400\n# citation_max_tokens = 8192\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax`, `qwen` (DashScope), and `deepseek` providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `30` | Exponential backoff base; delays are `min(base × 2^(attempt−1), 300)` seconds |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Emit `logger.warning` in server log when per-job ingest cost exceeds this threshold; job continues normally |\n| `cost.hard_gate_usd` | float | `2.00` | Permanently fail the ingest job (DEAD) and record a `cost_gate_exceeded` audit event when per-job cost exceeds this threshold |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × (backup_count + 1)` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n| `ingest.max_source_chars` | int | `32000` | Character limit applied to each source before the LLM call. Sources exceeding this limit are truncated; the page's `sources:` frontmatter entry gets `truncated: true` and lint emits a warning. Override per-run with `--max-source-chars N`. _(v1.0.0)_ |\n| `ingest.citation_source_lines` | int | `400` | Number of source lines the LLM sees during Pass 4 (citation annotation). Increase if lint reports `out_of_range` on long sources such as transcripts or PDFs. Raise `citation_max_tokens` in proportion when increasing this value. |\n| `ingest.citation_max_tokens` | int | `8192` | Output token budget for the annotated section returned by Pass 4. Increase when `citation_source_lines` is raised or wiki sections are long, to avoid truncated annotations. |\n| `query.context_wiki_pct` | float | `0.60` | Fraction of the model context window reserved for wiki source pages. _(v1.0.0)_ |\n| `query.context_history_pct` | float | `0.20` | Fraction reserved for conversation history. _(v1.0.0)_ |\n| `query.context_system_pct` | float | `0.15` | Fraction reserved for system prompt and instructions. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_index_pct` | float | `0.05` | Fraction reserved for the wiki index preamble. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_window` | int | _(auto)_ | Override the built-in context window lookup for the configured model. Use when running a local or custom model whose window size is not in the built-in table. _(v1.0.0)_ |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement is wired into the ingest pipeline: `Orchestrator._guard_ingest_cost()` is called after each ingest LLM call with the returned cost, before the job is marked complete.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | `logger.warning` in server log; job continues normally |\n| `hard_gate_usd` | $2.00 | Server/ingest path: records `cost_gate_exceeded` audit event + permanently fails job (DEAD). Interactive CLI: prompts `Proceed? [y/N]` |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `min(backoff_base_seconds × 2^(attempt − 1), 300)` \nWith the default base of 30 s: attempt 1 waits 30 s, attempt 2 waits 60 s, attempt 3 waits 120 s (all capped at 300 s). The `retry_after` timestamp is stored in the jobs table; `dequeue()` skips any job whose `retry_after` is still in the future. Applied only to transient errors (network timeouts, connection failures, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` by default. The bind address is configurable via `server.host` in `config.toml` (e.g. `host = \"0.0.0.0\"` to expose on all interfaces for LAN access). No built-in authentication — restrict via firewall when exposing beyond loopback. Remote access without a reverse proxy is not recommended on shared or networked machines.\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs and unassigned slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `-w ` / `--wiki ` to target a specific wiki.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; when all local source files are missing from disk the page is transitioned to `archived` (if only some sources are missing it is transitioned to `stale` instead). Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: lint evaluates all local (non-URL) source paths listed in the page's frontmatter. If **all** of them are missing from disk, the page is transitioned to `archived`. If only **some** are missing — meaning valid content from at least one source still exists — the page is transitioned to `stale` instead, preserving it for re-ingest rather than discarding it. _(v1.0.2: previously any single missing source triggered an immediate archive.)_\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Cascade link cleanup\n\nWhenever a page transitions to `archived` — whether triggered manually (`synthadoc lifecycle archive`), via the Obsidian Lifecycle modal, the MCP `synthadoc_lifecycle` tool, or automatically by lint detecting a missing source — Synthadoc immediately scans all other active pages and removes every `[[archived-slug]]` wikilink pointing to the now-archived page.\n\n- **Inline links** — `[[archived-slug]]` becomes plain text (the display text if an alias was used, otherwise the slug itself).\n- **List-item links** — a list item whose only content is `[[archived-slug]]` is dropped entirely.\n- **Archived pages are skipped** — pages already archived are not rewritten.\n- **System pages are skipped** — index, overview, dashboard, log, and purpose pages are excluded.\n\nThe cleanup runs synchronously before the response is returned, so the wiki is consistent immediately. The CLI and MCP responses include a `cascade_links_removed_from` field listing the affected slugs. During `lint run`, cascade is batched once after all auto-archive transitions complete, and the count is added to `dangling_links_removed` in the lint report.\n\nLint's existing periodic dangling-link cleanup (`lint run --scope orphans`) remains in place as a safety net for any dead links that predate this feature or arrive through other paths.\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [...], \"input_tokens\": 1240, \"output_tokens\": 380, \"tokens_used\": 1620}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […], \"input_tokens\": N, \"output_tokens\": N, \"tokens_used\": N}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Streaming Token Audit\n\nStreaming LLM responses do not return token counts in the same way as blocking calls. To track real usage in the audit trail:\n\n**Provider-side collection** — each `LLMProvider` subclass exposes two instance attributes: `last_stream_input_tokens` and `last_stream_output_tokens`. These default to `0` and are populated from the API-specific usage field when the stream ends:\n\n| Provider | Mechanism | Verified |\n|---|---|---|\n| OpenAI | `stream_options={\"include_usage\": True}` passed to `chat.completions.create()`; final chunk carries `chunk.usage.prompt_tokens` / `completion_tokens` with empty `choices` | ✅ |\n| Anthropic | `message_start` event → `event.message.usage.input_tokens`; `message_delta` event → `event.usage.output_tokens` | ✅ |\n| Ollama | Final chunk with `done=True` → `prompt_eval_count` / `eval_count` | ✅ |\n| DeepSeek | Same `OpenAIProvider` path as OpenAI; DeepSeek's OpenAI-compatible API supports `stream_options` | ✅ |\n| MiniMax (reasoning: M2.5+) | Detects `` in stream → falls back to blocking `complete()` → captures exact counts from `resp.usage` | ✅ |\n| MiniMax (non-reasoning, e.g. M3 thinking=disabled) | Same `OpenAIProvider` path; MiniMax API silently ignores `stream_options`. Falls back to character-based estimate (÷ 3.5 chars/token) from prompt + answer lengths. Accuracy ±20%. | ✅ estimated |\n| Gemini | Same `OpenAIProvider` path via `generativelanguage.googleapis.com/v1beta/openai/`; Google's compatibility layer honours `stream_options` — verified live (Gemini 2.5 Flash Lite, 50K tokens) | ✅ |\n| Groq | Same `OpenAIProvider` path; Groq's OpenAI-compatible API supports `stream_options` | ✅ |\n| Qwen (DashScope) | Same `OpenAIProvider` path via DashScope; `stream_options` honoured — verified live (qwen-plus, 28K tokens) | ✅ |\n| Qwen (Ollama) | Ollama path → `prompt_eval_count` / `eval_count` | ✅ |\n\n**Forwarding to `done` event** — after `complete_stream()` is exhausted, `QueryAgent.run_stream()` reads the provider attributes and includes them in the final `done` event payload: `input_tokens`, `output_tokens`, `tokens_used`.\n\n**Audit record** — `Orchestrator.query_stream()` consumes the `done` event and calls `estimate_cost(model, input_tokens, output_tokens, is_local)` from `synthadoc/providers/pricing.py`, then calls `AuditDB.record_query()` with the real token count and computed cost. The audit row at `GET /audit/queries` will therefore always show non-zero values for cloud providers.\n\nWhen a provider does not report streaming usage (e.g. a future provider not yet wired), the attributes stay at `0`; the audit row records `tokens=0, cost_usd=0.0` rather than crashing, matching the pre-fix behaviour.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 5) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 5 # turns to include in each request (default: 5; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp, cascade_links_removed_from: [str]}` or `{error, cascade_links_removed_from: []}` _(cascade field added v1.0.2)_ | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions for Codex CLI / OpenCode (if present)\n├── CLAUDE.md ← same content; loaded by Claude Code (if present)\n├── GEMINI.md ← same content; loaded by Gemini CLI (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── hooks/ ← user hook scripts (always included)\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ ├── extracted/ ← text sidecars + PDF pagemaps (always included)\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `logs/`.\n\n> **v0.9.3 fix:** `hooks/` and `.synthadoc/extracted/` were previously missing from backups. Omitting `extracted/` caused the Source Viewer and Provenance citation modal to show \"Could not read source file\" after restore.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## 29. Pre-LLM Source Sanitizer\n\nEvery source document passes through a sanitizer immediately after text extraction, before any LLM call. The sanitizer removes six categories of content that could manipulate the LLM or degrade compilation quality:\n\n| Category | Action | Warning logged? |\n|---|---|---|\n| Zero-width characters (U+200B, U+200C, U+200D, U+FEFF) | Removed silently | No |\n| Bidi override characters (U+202A–U+202E, U+2066–U+2069) | Removed | Yes |\n| HTML comments (``) | Removed silently | No |\n| Hidden CSS spans (`display:none`, `visibility:hidden`) | Removed silently | No |\n| Base64 blobs ≥ 200 consecutive characters | Replaced with `[base64 content removed]` | Yes |\n| Instruction-override phrases (8 patterns: \"ignore previous instructions\", \"disregard the above\", \"override your system prompt\", etc.) | Replaced with `[redacted]` | Always |\n\nWhen content is removed and a warning is appropriate, the server logs a `WARN` entry:\n```\n[WARN] sanitizer stripped content from 'papers/survey.pdf': bidi overrides, instruction-override phrase\n```\n\nThis is a pure pre-processing step with zero LLM cost. Truncation (if configured) is applied after sanitization, so the character budget is consumed by clean text only.\n\n---\n\n## 30. Per-Source Truncation Flag\n\nLarge source documents are truncated before they reach the LLM. By default the limit is **32,000 characters** (~8,000 tokens), raised from the prior hardcoded 8,000-character limit. When a source exceeds this limit, its compiled wiki page records a `truncated: true` flag in its `sources:` frontmatter entry.\n\n### Configuration\n\n```toml\n[ingest]\nmax_source_chars = 32000 # default; raise for large PDFs or books\n```\n\n### Per-ingest override\n\nThe limit can be raised for a single ingest run without modifying the config:\n\n```bash\nsynthadoc ingest papers/large-textbook.pdf --max-source-chars 128000\n```\n\nThe same `max_source_chars` field is accepted by the HTTP `POST /jobs/ingest` body and the MCP `synthadoc_ingest` tool.\n\n### Lint surfacing\n\n`synthadoc lint` emits a warning for any page that has a truncated source:\n\n```\n[WARN] quantum-computing.md: source 'papers/large-survey.pdf' was truncated at ingest\n (source exceeded max_source_chars=32000 — 87,412 chars in source).\n To re-ingest with a higher limit (this source only):\n synthadoc ingest papers/large-survey.pdf --max-source-chars 128000\n To raise the limit for all future ingests:\n set [ingest] max_source_chars = 128000 in your config\n```\n\n---\n\n## 31. Proportional Context Budget\n\nQuery context is allocated proportionally to the configured model's context window, replacing the prior hard cap. The default allocation:\n\n| Slice | Default | Purpose |\n|---|---|---|\n| `context_wiki_pct` | 60% | Wiki page content (ranked by BM25 + vector score) |\n| `context_history_pct` | 20% | Chat turn history (newest-first, oldest dropped when over budget) |\n| `context_system_pct` | 15% | System prompt and purpose document |\n| `context_index_pct` | 5% | `ROUTING.md` and search index |\n\nPercentages must sum to ≤ 100. Configure in `[query]`:\n\n```toml\n[query]\ncontext_window = 0 # 0 = auto-detect from known model map\ncontext_wiki_pct = 60\ncontext_history_pct = 20\ncontext_system_pct = 15\ncontext_index_pct = 5\n```\n\n> **Note:** `context_system_pct` and `context_index_pct` are parsed and validated but not yet enforced as hard caps in this release — the system prompt uses a conservative fixed limit, which is well within the configured 15% slice for all supported models. Full per-slice enforcement is planned for v1.1.\n\n### Model context window map\n\nSynthadoc ships a built-in prefix-matched table:\n\n| Model | Context window |\n|---|---|\n| `claude-opus-4*`, `claude-sonnet-4*`, `claude-haiku-4*` | 200,000 tokens |\n| `gpt-4o*`, `gpt-4-turbo*` | 128,000 tokens |\n| `gpt-4` (exact) | 8,192 tokens |\n| `gpt-3.5-turbo*` | 16,385 tokens |\n| Unknown / fallback | 128,000 tokens |\n\nSet `context_window = N` in config to override the map (useful for local Ollama models with non-standard context sizes).\n\nAt the defaults on a 200k-token model: wiki slice ≈ 480,000 chars — far more than any practical query needs. On constrained models, the budget scales down automatically so the same config works across model families.\n\n---\n\n## 32. Knowledge Graph\n\nSynthadoc computes a wikilink graph across all active and draft pages during every lint run. The graph is stored in two tables in `audit.db`:\n\n```sql\nCREATE TABLE graph_nodes (\n slug TEXT PRIMARY KEY,\n cluster_id INTEGER NOT NULL,\n updated_at TEXT NOT NULL\n);\n\nCREATE TABLE graph_edges (\n from_slug TEXT NOT NULL,\n to_slug TEXT NOT NULL,\n weight INTEGER NOT NULL DEFAULT 1,\n edge_type TEXT NOT NULL DEFAULT 'mixed',\n PRIMARY KEY (from_slug, to_slug)\n);\n```\n\nEdge weight is a composite signal from two sources:\n\n| Edge type | Source | Weight contribution |\n|-----------|--------|---------------------|\n| `wikilink` | `[[slug]]` occurrences in page body | +1 per occurrence |\n| `co_source` | pages compiled from the same source file (matched by hash) | +2 per shared source |\n\nMost edges are `mixed` (have both types). Pure `co_source` edges surface hidden relationships — pages compiled from the same source document are linked immediately after ingest, before any wikilinks are created. `edge_type` is stored on each edge so the web UI can render co-source edges with a different visual style.\n\nCluster IDs are assigned by Louvain community detection — pages that link densely to each other end up in the same cluster.\n\n### REST API\n\n**`GET /graph`** — returns the full graph with per-node enrichment:\n\n```json\n{\n \"status\": \"ready\",\n \"node_count\": 45,\n \"edge_count\": 123,\n \"cluster_count\": 4,\n \"nodes\": [\n {\"slug\": \"quantum-error-correction\", \"title\": \"Quantum Error Correction\",\n \"type\": \"concept\", \"state\": \"active\", \"cluster_id\": 2}\n ],\n \"edges\": [\n {\"from\": \"quantum-error-correction\", \"to\": \"shor-algorithm\", \"weight\": 3, \"edge_type\": \"mixed\"}\n ]\n}\n```\n\nWhen the graph tables are empty (first call after upgrade, or before the first lint run), the server returns `{\"status\": \"computing\"}` and starts a background build immediately. The web UI polls every 2 seconds and renders the graph when the status becomes `\"ready\"`.\n\n### Web UI\n\nThe **Graph** tab (alongside Chat in the top nav) renders a D3.js force-directed graph. Nodes are colored by cluster. Edge thickness is proportional to weight (1–4 px). Pure `co_source` edges (no wikilink) are rendered with a dashed stroke, making hidden source-based relationships visually distinct from explicit cross-links. Clicking a node opens a sidebar showing the page title, type and lifecycle state badges, cluster assignment, and an **\"Ask about this →\"** button that switches to the Chat tab with a pre-filled query.\n\nControls: zoom in/out (scroll or pinch), drag nodes, filter by page type.\n\n### System page exclusion _(v1.1.0)_\n\nThe five structural system pages — `overview`, `index`, `dashboard`, `purpose`, and `log` — are excluded from both `graph_nodes` and `graph_edges`. These pages link to nearly every other page and would act as false hubs, inflating edge counts and distorting Louvain cluster detection. System page exclusion is applied inside `_build_graph()` before any edge computation.\n\n### Obsidian knowledge graph panel _(v1.1.0)_\n\n→ Full description: [§33 Multi-Platform Agent Skill Files](#33-multi-platform-agent-skill-files) covers scaffold; the graph panel itself is documented in [§8 Obsidian Plugin](#8-obsidian-plugin) command table.\n\nThe Obsidian plugin includes a **draggable modal** knowledge graph panel (`Graph: show knowledge graph` command). It fetches from the same `GET /graph` endpoint as the web UI and renders a force-directed Canvas 2D simulation using Verlet integration (no external library). System pages are already excluded from the server response.\n\n| Feature | Obsidian panel | Web UI Graph tab |\n|---------|----------------|-----------------|\n| Engine | Canvas 2D + Verlet | D3.js SVG |\n| Node click | Opens the wiki page in the current pane | Opens the node sidebar with \"Ask about this →\" |\n| **Best for** | **Navigating to and editing pages** | **Asking questions about a page** |\n| Draggable panel | Yes — drag header to reposition | Fixed tab layout |\n| Node cap | 300 most-connected (see note) | No cap |\n| Cluster legend | Yes | Yes |\n| Type filter | Yes | Yes |\n\n**Rule of thumb:** Use the **web UI** when you want to ask questions about a node. Use the **Obsidian panel** when you want to navigate to and edit a page.\n\n**Node cap rationale:** The Obsidian panel uses a hand-written Verlet integrator with O(n²) pairwise charge repulsion — every node repels every other node on each animation tick. At 300 nodes that is ~90,000 operations per frame, which runs smoothly at 60 fps inside Obsidian's Electron renderer. At 500 nodes it is ~250,000 operations per frame and starts to lag noticeably. The web UI has no cap because D3's `forceManyBody` uses a Barnes-Hut tree approximation (O(n log n)), which scales to thousands of nodes without frame-rate issues. When the cap is hit, nodes are ranked by degree and the 300 most-connected are shown; a banner reports the total page count and the number of hidden nodes, and suggests using the type filter to bring the set below 300.\n\n→ User walkthrough: [Quick-Start Guide §24 — Knowledge Graph](docs/user-quick-start-guide.md#knowledge-graph)\n\n---\n\n## 33. Multi-Platform Agent Skill Files\n\n\n\n`synthadoc init` and `synthadoc scaffold` write three agent skill files at the wiki root:\n\n| File | Format | Read by |\n|------|--------|---------|\n| `AGENTS.md` | OpenAI Agents SDK / Codex / OpenCode | Codex CLI, OpenCode, generic agents |\n| `CLAUDE.md` | Claude Code / Anthropic | Claude Code (loaded automatically when the wiki folder is opened) |\n| `GEMINI.md` | Gemini CLI | Gemini CLI |\n\nAll three files carry **identical content**: a title block, the LLM-generated domain guidelines for the wiki, a quick-reference CLI command table, and an MCP tool table. The only difference is the heading comment. This means any AI coding tool that opens the wiki root gets full Synthadoc guidance without manual setup.\n\n### Content generated by `scaffold`\n\nEach skill file is generated from the same LLM call that produces `purpose.md` and `index.md`. The LLM receives the current wiki page list and domain description, then generates:\n\n1. **Domain label** — a precise human-readable name inferred from wiki content (e.g. `History of Computing`, not `My Wiki`)\n2. **Domain guidelines** — 3 domain-specific ingest and query guidelines as plain sentences\n3. **Quick Reference table** — pre-filled with the current wiki name and port\n\nIf the wiki was created before v1.1.0, running `synthadoc scaffold` once regenerates all three files and adds `CLAUDE.md` and `GEMINI.md` to any wiki that only had `AGENTS.md`.\n\n### Template source\n\nThe instruction body is defined in `synthadoc/cli/_init.py` as `_AGENT_INSTRUCTION_BODY` and shared by all three templates, keeping the files permanently in sync. The scaffold agent substitutes `{domain}`, `{guidelines}`, and `{port}` at write time.\n\n---\n\n## 34. Synthadoc Scaffold Marker\n\n\n\nThe scaffold marker `` (written as an HTML comment) separates **user-authored content** (preserved) from **scaffold-managed content** (rewritten on each `synthadoc scaffold` run).\n\n### Single-marker mode (index.md)\n\nThe marker appears once, below the H1 heading in `index.md`. Everything above the marker is the user zone; everything below is replaced with new LLM-generated categories.\n\n```markdown\nMy custom introduction — maintained by hand.\n\n\n\n## Pioneers and Visionaries\n- [[alan-turing]]\n...\n```\n\nIf the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### Per-section multi-marker mode (purpose.md)\n\n`purpose.md` uses a **per-section** approach: the marker appears inside each `## Section` heading. Within a section, content the user writes **above** the marker is preserved; content below is replaced with fresh LLM output. Sections without a marker in the existing file are replaced entirely (and a marker is added). Sections in the existing file that the scaffold template does not include are kept unchanged.\n\n```markdown\n## Overview\n\nMy custom overview note.\n\n\n\nLLM-generated overview content goes here.\n\n## What Belongs\n\n\n\nLLM-generated scope definition.\n```\n\nThis design lets users annotate any section of `purpose.md` without losing their notes on the next scaffold run.\n\n### Marker preservation guarantee\n\n- `preserve_user_zone()` in `scaffold_agent.py` handles both modes.\n- If a section has no marker in the existing file, `preserve_user_zone()` treats the entire section body as user content and preserves it, replacing only what falls below any marker it finds.\n- The scaffold NEVER touches `config.toml`, `dashboard.md`, or any wiki page (`wiki/*.md`).\n\n→ User walkthrough: [Quick-Start Guide §14 — Enrich the wiki with scaffold](docs/user-quick-start-guide.md#scaffold)\n\n---\n\n## Customization\n\nSynthadoc exposes four extension points — all hot-loaded, no server restart required.\n\n### Custom skills (new file formats)\n\nSubclass `BaseSkill` (Apache-2.0 — no AGPL obligation on your skill code), drop the folder in `/skills/` or `~/.synthadoc/skills/`, and Synthadoc picks it up on the next ingest. Skills match by file extension or intent prefix and support any Unicode text, including CJK prefixes.\n\n→ Full interface, manifest format, and examples: [§5 Skills System](#5-skills-system) and [§17 Plugin Development Guide](#17-plugin-development-guide)\n\n### Custom LLM providers\n\nSubclass `LLMProvider` from `synthadoc/providers/base.py` (Apache-2.0) and place the file in `~/.synthadoc/providers/` or the wiki `providers/` directory. Switch active provider with one config line.\n\n→ Full interface and wiring instructions: [§10 Configuration — Provider switching](#10-configuration)\n\n### Hooks\n\nShell scripts (any language) that fire on `on_ingest_complete` and `on_lint_complete`. Receive a JSON context on stdin. Set `blocking = true` to gate the operation on the hook's exit code — useful for CI pipelines and quality gates.\n\n→ Full event schema, context JSON, and blocking behaviour: [§11 Hook System](#11-hook-system)\n\n### Cache control\n\nThree cache layers (embedding, LLM response, provider prompt cache) invalidate automatically on source-file change (SHA-256). Force a fresh call with `--force`, or wipe all cached responses with `synthadoc cache clear -w `.\n\n→ Layer-by-layer breakdown and invalidation rules: [§12 Cache System](#12-cache-system)\n\n### Multi-Platform Agent Skill Files\n\nEvery Synthadoc wiki ships three companion files that give AI coding agents a complete, self-contained operating guide for the wiki:\n\n| File | Platform |\n|---|---|\n| `AGENTS.md` | Codex CLI, OpenCode, and any agent that reads the AGENTS.md convention |\n| `CLAUDE.md` | Claude Code (this session's tool) — highest-priority instruction source in Claude's hierarchy |\n| `GEMINI.md` | Gemini CLI — relevant because Synthadoc's default LLM provider is `gemini-2.5-flash-lite` |\n\nAll three files share identical body content generated from the same template; they differ only in their H1 heading so each agent recognises its own file first. The body covers:\n\n- **Domain guidelines** — LLM-generated bullet list of domain-specific ingest and query rules (produced by `scaffold`)\n- **Quick reference table** — every common CLI command with the correct `synthadoc` syntax\n- **Server startup** — how to start the server and verify it is running\n- **Ingest examples** — local file, URL, YouTube, agent session `.jsonl`, force reingest, dry-run\n- **Query** — basic and streaming query syntax; citation marker format\n- **Lint** — when to run and what each check does\n- **Lifecycle** — transition commands with state names\n- **Page schema** — frontmatter structure with all fields\n- **MCP tools table** — all 12 tools with purpose descriptions\n\n**How the files are created and updated:**\n\n1. `synthadoc init` — writes all three files with default domain guidelines and the wiki's configured port.\n2. `synthadoc scaffold` — regenerates all three with LLM-produced domain guidelines derived from the current wiki state. The `scaffold` command prints a confirmation line for each file.\n\n**Use cases:**\n\n- Open Claude Code in a Synthadoc wiki directory and it immediately knows how to ingest, query, lint, and manage lifecycle — no additional setup. Claude Code can also start the server in background mode (`synthadoc serve -w -b`), stop it cleanly, and orchestrate multi-step workflows, effectively acting as a control plane for the wiki.\n- Gemini CLI reads `GEMINI.md` and can run `synthadoc` commands with the correct flags without asking the user for help.\n- CI pipelines with Codex CLI use `AGENTS.md` to drive automated lint and lifecycle transitions on PR merge.\n- Agent session files produced by Claude Code (`.jsonl` under `~/.claude/projects/`) can themselves be ingested into the wiki, capturing the reasoning behind design decisions alongside the source documents.\n\n→ See [README.md — Interfaces & Integration](../README.md#interfaces--integration) for the feature comparison row and quick-start instructions.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v1.1.2\n\n- **Exponential backoff on transient URL fetch failures** — jobs failing with transient errors (connection timeouts, 5xx responses, dropped TCP connections including `RemoteProtocolError`) now re-queue with exponential backoff: `min(base × 2^(attempt−1), 300)` seconds (default base 30 s). Previously all failures retried immediately, exhausting the retry budget within seconds. `dequeue()` skips any job whose `retry_after` timestamp is still in the future; other jobs run normally during the wait window. See [§14 Job Queue](#14-job-queue).\n- **Accurate retry outcome in job warning logs** — the transient-failure warning now reads the current attempt count from the database before logging, so it correctly reports `will retry` vs. `final attempt` (was always `will retry` regardless of retry state).\n- **Scheduler subprocess timeout** — scheduled jobs (`lint`, `ingest --batch`, etc.) are now guarded by `asyncio.wait_for(proc.communicate(), timeout=job_timeout_seconds)`; a hung subprocess is killed after the timeout and the run is recorded as `failed`. Previously a stalled subprocess could block the scheduler task indefinitely. `job_timeout_seconds` defaults to 600 s and is now shown (uncommented) in fresh-init `config.toml`.\n- **Streaming query token and cost accuracy** — the streaming query audit record was written with `tokens=0, cost_usd=0.0`; it now records the actual LLM token usage and cost from the streamed response.\n- **Cost hard gate timing** — the cost hard gate now fires before LLM tokens are spent and pages are written; previously the gate ran after the LLM call completed, allowing partial page writes before rejection.\n- **Session state memory growth bounded** — `_session_state` in the HTTP server is pruned at 100 concurrent sessions (LRU eviction) to prevent unbounded memory growth on long-running servers.\n- **Streamed `` block cleanup** — a leaked fallback asyncio task is now cancelled when a streaming response ends inside an unclosed `` block (e.g. reasoning-model responses cut off mid-think).\n- **Config validation for staging fields** — `staging_policy` values other than `off`, `all`, and `threshold`, and non-numeric `staging_confidence_min`, were silently accepted and caused runtime errors; they now raise a `ConfigError` at server startup.\n- **Job status guard on retry** — retrying a job that is already `in_progress`, `done`, or `skipped` now returns an error instead of silently resetting it to `pending`.\n- **Jobs endpoint performance** — `GET /jobs/{id}` and `DELETE /jobs/{id}` now use a direct primary-key lookup instead of a full table scan; eliminates O(N) reads on large job queues.\n- **Vector migration GC safety** — the background vector migration `asyncio.Task` is now held in a strong reference to prevent premature garbage collection mid-run.\n- **Quota error stack trace suppression** — Qwen 403 errors and Claude Code session-limit exhaustion now emit a concise warning naming the actionable config key, instead of a full stack trace.\n- **Scaffold agent JSON robustness** — `MiniMax-M3` and other reasoning models can truncate mid-JSON; the scaffold agent now handles truncated JSON output gracefully rather than failing the job.\n- **Staging/Candidates section in agent skill files** — `AGENTS.md`, `CLAUDE.md`, and `GEMINI.md` generated by `synthadoc scaffold` now include a `## Staging and Candidates` section.\n- **Citation failure pagination fix** — `GET /audit/citation-failures` was applying the `LIMIT` in SQL before filtering by `page_slug`, returning incorrect results for paginated per-page queries; filtering now runs before the limit.\n\n### v1.1.0\n\n- **Weighted knowledge graph edges** — graph edges now carry two signals: wikilink occurrences (+1 each) and co-source connections — pages compiled from the same source file (matched by SHA-256 hash) — (+2 per shared source). `edge_type` field added to `graph_edges` table (`wikilink`, `co_source`, or `mixed`). `GET /graph` exposes `edge_type` per edge. Web UI renders edge thickness proportional to weight (1–4 px) and dashed lines for pure co-source edges. Co-source edges appear immediately after ingest — before any wikilinks — surfacing hidden relationships in the graph. Schema migrated to version 3.\n- **Multi-platform agent skill files** — `synthadoc init` and `synthadoc scaffold` write `CLAUDE.md`, `AGENTS.md`, and `GEMINI.md` at wiki root; all three carry LLM-generated domain guidelines and the full CLI quick-reference so Claude Code, Codex CLI, OpenCode, Pi, and Gemini CLI users get first-class guidance without manual setup. Run `synthadoc scaffold` once on any pre-v1.1.0 wiki to generate the new files. See [§33](#33-multi-platform-agent-skill-files).\n- **Session history ingestion skill** — `.jsonl` session files from Claude Code and Codex CLI are ingested as wiki pages; human turns and substantive assistant responses extracted; tool calls, thinking blocks, and sub-agent scaffolding skipped; format auto-detected; `suggested_slug` derived from session date and first user message. Converts ephemeral problem-solving knowledge into searchable wiki entries. See [§5 Skills System — Session Skill](#session-skill--ai-session-history-ingestion).\n- **Obsidian knowledge graph panel** — new `Graph: show knowledge graph` command in the Obsidian plugin opens a draggable modal with a Canvas 2D force-directed graph. Features: Verlet integration, type filter dropdown, cluster legend, hover tooltip (title, slug, type, state, cluster, connections), click-to-open page in current pane, node drag to repin. Closes only via the × button — Escape and outside-click do not close the panel, matching the behaviour of other Synthadoc modals. **Large-wiki cap:** when the node set exceeds 300 (after any type filter is applied), nodes are ranked by degree (total edge count) and the 300 most-connected are displayed; edges to capped-out nodes are removed; a banner explains the cap and suggests using the type filter to narrow the view below 300. See [§8 Obsidian Plugin](#8-obsidian-plugin) and [§32 Knowledge Graph — Obsidian panel](#obsidian-knowledge-graph-panel-v110).\n- **Per-section scaffold markers in `purpose.md`** — `purpose.md` now uses per-section `` markers. Users can write content above the marker in any `## Section` block; that content is preserved on every re-scaffold. A single-marker mode for `index.md` continues to work as before. See [§34 Synthadoc Scaffold Marker](#34-synthadoc-scaffold-marker).\n- **System pages excluded from knowledge graph** — `overview`, `index`, `dashboard`, `purpose`, and `log` are filtered out of `_build_graph()` before any edge computation so they cannot act as false hubs in Louvain cluster detection. Both the Obsidian graph panel and the web UI Graph tab reflect this exclusion.\n- **Web UI graph cluster legend** — the web UI Graph tab now shows a cluster legend below the canvas (matching the Obsidian plugin panel), with a color swatch and label for each Louvain cluster present in the filtered node set.\n- **`_merge_section_into_page` deduplication** — force re-ingest of the same source file (e.g. an updated `.jsonl` session) now replaces the page body rather than appending a duplicate section. `_files_match()` normalizes absolute vs. relative paths so the same file is recognized regardless of how it was passed to the ingest command.\n- **`purpose.md` domain label inference** — `scaffold` now infers a precise `domain_label` from current wiki content (e.g. `History of Computing`) instead of using a generic placeholder; the label is used in all three agent skill files and in `purpose.md`'s Overview section.\n- **CI: Windows runner fix** — `scripts/sync_plugin.py` and `scripts/sync_web_ui.py` now use `shutil.copy2` instead of `cp -f`, resolving CI failures on Windows runners for the sync check.\n\n### v1.0.2\n\n- **Cascade link cleanup on archive** — archiving any page immediately removes every `[[slug]]` wikilink pointing to it from all other active pages, without waiting for the next `lint run`. Inline links become plain text; list-item-only links are dropped entirely. The HTTP `POST /lifecycle/transition` and MCP `synthadoc_lifecycle` responses now include a `cascade_links_removed_from` field listing the rewritten page slugs. During `lint run`, cascade runs once after all auto-archive transitions complete and the count is included in the lint report's `dangling_links_removed` field. System pages (index, dashboard, purpose, log, overview) and already-archived pages are skipped. The existing dangling-link lint pass remains as a safety net for dead links that predate this release.\n- **Multi-source partial-archive fix** — when a page has multiple local source files and only some are missing from disk, lint now transitions the page to `stale` (not `archived`), preserving it for re-ingest. The page is only archived when every local source file is gone. URL sources are not affected by this check.\n\n### v1.0.1\n\n- **Scaffold JSON extraction hardened** — replaced the greedy regex used to extract the LLM's JSON response with a brace-balanced parser; eliminates false-positive matches on `[[wikilink]]` syntax that previously caused `json.loads()` failures and silent scaffold corruption\n- **Generated file validation** — scaffold now validates every LLM-generated file (index, purpose, AGENTS.md) for minimum length, required frontmatter, and `[[wikilink]]` presence before writing to disk; validation failure marks the job as failed with a structured error message rather than silently writing a malformed page\n- **ROUTING.md regeneration validation** — `routing init` and scheduled routing regeneration now validate the regenerated file for minimum line count and anchor presence before applying it; prevents a corrupt or truncated ROUTING.md from entering the wiki\n- **Scaffold failure reporting** — scaffold skip and failure reasons are now surfaced in the terminal output and job error field; previously silent failures (e.g. LLM timeout, JSON parse error) are now visible without tailing the JSON log\n- **Install-time scaffold removed** — `synthadoc install` no longer runs the LLM scaffold call during installation; scaffold is a separate step (`synthadoc scaffold`) that the user runs after setting their API key, avoiding confusing API-key-not-set errors at install time\n- **AquaFlow LLM evaluation** — 15-question M&A due-diligence benchmark across five models (MiniMax-Think M3, Claude Opus 4.8, Claude Sonnet 4.6, DeepSeek-R1, Qwen Plus); full report at `docs/example/aquaflow/evaluation/report/llm-query-benchmark.md`\n\n### v1.0.0\n\n- **Pre-LLM source sanitizer** — every source document passes through a sanitizer before any LLM call; removes six categories of potentially harmful content (zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars, instruction-override phrases); a `WARN` log entry is emitted when bidi overrides or instruction-override phrases are stripped; zero LLM cost; applied before truncation so the character budget is consumed by clean text only\n- **Per-source truncation flag** — default character limit raised from 8,000 to 32,000 characters per source (~8,000 tokens); truncated sources are flagged with `truncated: true` in the page's `sources:` frontmatter; `--max-source-chars N` CLI flag and matching `POST /jobs/ingest` body field and MCP `synthadoc_ingest` parameter override the limit per run; `synthadoc lint` emits a warning for any page with a truncated source and suggests the override command; configurable via `[ingest] max_source_chars` in `config.toml`\n- **Proportional context budget** — query context is allocated proportionally to the configured model's context window; four configurable slices: `context_wiki_pct` (60%), `context_history_pct` (20%), `context_system_pct` (15%), `context_index_pct` (5%); built-in prefix-matched model context window table covers Claude 4, GPT-4o/turbo/3.5, and GPT-4; unknown models fall back to 128,000 tokens; `context_window = N` in `[query]` overrides the table for local or custom models\n- **Knowledge graph** — wikilink graph computed across all active and draft pages during every lint run; stored in `graph_nodes` and `graph_edges` tables in `audit.db`; edge weight = number of `[[wikilink]]` references between two pages; cluster IDs assigned by Louvain community detection; `GET /graph` REST endpoint returns nodes with per-node enrichment (title, type, state, cluster) and edges; first call after upgrade triggers a background build and returns `{\"status\":\"computing\"}`; web UI **Graph** tab (D3.js force-directed graph) — nodes colored by cluster, click to see page details and an \"Ask about this →\" button that opens a pre-filled chat query\n- **Citation accuracy & rendering improvements** — six correctness fixes to the ingest citation pipeline: (1) full source text passed to the decision LLM so citations can reference any line in the document; (2) `_annotate_citations` bugs A–F resolved (false-positive matches, off-by-one line ranges, duplicate markers, escaped bracket handling, multi-paragraph span errors, and source filename normalisation); (3) `_has_citations` check added — lint warns and suggests a model upgrade when a page has zero citation markers after synthesis; (4) `Key Data` section extraction hardened so numerical facts, rates, and formulas are preserved verbatim across the annotation pass; (5) `obsidianCitationsToGfm` rendering fixed in the web UI — citation superscripts now display correctly in the chat panel alongside streamed answers; (6) reading view set as the default Obsidian display mode for wiki pages\n\n### v0.9.3\n\n- **Backup now includes `hooks/` and `.synthadoc/extracted/`** — both directories were silently omitted from backups. `hooks/` contains user-customised hook scripts; `.synthadoc/extracted/` contains per-source text sidecars and PDF pagemaps written by the ingest agent. Without these, the Source Viewer and Provenance citation modal showed \"Could not read source file\" after restore.\n- **PyPI publish workflow** — `publish.yml` triggers on GitHub Release, builds the wheel with hatchling, and uploads to PyPI via OIDC trusted publishing (no API token stored). `pip install synthadoc` now works.\n- **Web UI bundled in pip package** — `synthadoc/data/web-ui/dist/` is now included in the wheel. `synthadoc web` previously returned 503 on pip-installed instances because the React dist was not packaged.\n- **CI sync checks** — PRs now fail if `synthadoc/data/obsidian-plugin/`, `synthadoc/data/web-ui/`, or `README-pypi.md` are out of sync with their source. All sync-check steps use `shell: bash` for Windows runner compatibility.\n- **`bump_version.py` extended** — now also updates `synthadoc/data/obsidian-plugin/manifest.json` alongside the source manifest, preventing version drift between the bundled copy and the plugin source.\n\n### v0.9.2\n\n- **`synthadoc/data/web-ui/dist/` packaged** — web UI dist files committed to the repo and bundled in the wheel via hatchling artifacts.\n- **`scripts/sync_web_ui.py`** — helper script to sync `web-ui/dist/` into `synthadoc/data/web-ui/dist/` after `npm run build`.\n\n### v0.9.1\n\n- **PyPI metadata** — added `readme`, `keywords`, `classifiers`, and `[project.urls]` to `pyproject.toml` for correct PyPI rendering.\n- **`README-pypi.md`** — generated from `README.md` by `scripts/prepare_pypi_readme.py`; strips `pypi-strip` blocks and rewrites relative links to absolute GitHub URLs.\n- **Branch protection compatibility** — removed post-merge `build-plugin` CI job that pushed compiled plugin directly to `main`. Plugin sync is now enforced in PRs: CI fails if `synthadoc/data/obsidian-plugin/` is out of sync.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 5). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n"}, "2cf9bce407735ecd5d6c7ad8fc1aa4a8c6bd6753": {"t": 1785767248.3204632, "n": 219925, "h": "d6af8177ad1b082a", "c": "# Synthadoc — Design Document\n\n**Version:** 1.1.0 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n29. [Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer)\n30. [Per-Source Truncation Flag](#30-per-source-truncation-flag)\n31. [Proportional Context Budget](#31-proportional-context-budget)\n32. [Knowledge Graph](#32-knowledge-graph)\n33. [Multi-Platform Agent Skill Files](#33-multi-platform-agent-skill-files)\n34. [Synthadoc Scaffold Marker](#34-synthadoc-scaffold-marker)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for Codex CLI, OpenCode, and generic OpenAI agents\n CLAUDE.md ← same content as AGENTS.md; loaded automatically by Claude Code\n GEMINI.md ← same content as AGENTS.md; loaded automatically by Gemini CLI\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the full source text (bounded by `max_source_chars`) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `flag` / `skip`). Pages with `status='active'` are protected — sources that conflict with them trigger `flag` rather than `update` (RULE 1b).\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Orchestrator calls `_guard_ingest_cost()` on the returned cost: soft warn → `logger.warning` only; hard gate → `cost_gate_exceeded` audit event + `fail_permanent` (job → DEAD). On success, job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 0b — Key Data Extraction | None (regex) | Deterministic pre-processor: extracts numbers, percentages, currency amounts, ratios, formulas, underscore identifiers, and date ranges from source text via regex; appends them as a `[Key Data — extracted by pre-processor]` section before any LLM sees the text. Zero token cost; cannot hallucinate. Anchors the synthesis LLM to all quantitative candidates. |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads the full source text (bounded by `max_source_chars`) + BM25 candidates + `purpose.md` scope + status of each candidate page. Outputs per-page action: `create`, `update`, `flag`, `skip`. **RULE 1b:** a page with `status='active'` is human-reviewed and authoritative — when the source provides a conflicting value, date, formula, or conclusion, the action must be `flag`, not `update`. The only exception is a source that adds a topic the page does not yet mention at all. Decision cache version: `v2`. |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. If no citation markers are present in the returned body, a `citation_pass4_no_markers` WARNING audit event is written — the page is still saved but the condition is flagged for re-ingest with a more capable model. Zero-citation results are not cached so a subsequent re-ingest can succeed. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic. **Suppressed when TF fallback was used** (a small corpus with broad coverage may legitimately return fewer than 3 candidates — that is expected, not a gap).\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n| Citation presence (Check 5b) | Page body has ≥ 50 words and zero `^[filename:L-L]` citation markers → WARNING. Diagnostic only — does not block promotion to `active`. Indicates the annotation pass failed to produce markers, usually a model-compatibility issue (use Gemini 2.5 Flash or higher for reliable citation annotation). |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n### ActionAgent\n\nDispatches action-intent queries from the chat UI (e.g. \"activate a draft page\", \"show lint report\", \"archive a stale page\", \"run scaffold\"). Parses the user's free-text intent into a structured action via an LLM extraction prompt, then executes the action against the wiki and returns a human-readable result. When the intent is ambiguous (e.g. \"activate a page\" without specifying which), sets `needs_clarification=True` and returns a prompt and candidate list for the web UI to render as chip buttons.\n\n### ScaffoldAgent\n\nGenerates and updates the wiki's structural pages. Reads all current wiki pages and asks the LLM to organise them into 5–8 domain-appropriate categories, then writes `wiki/index.md` (category index with wikilinks), `wiki/purpose.md` (scope statement), and `AGENTS.md` (ingest and query guidelines). Also stamps a `categories:` field on each page's YAML frontmatter. When `ROUTING.md` already exists, regenerates it to stay in sync with the updated index structure.\n\n### RewriteAgent\n\nRewrites follow-up questions in multi-turn conversations into self-contained, standalone form before BM25 retrieval. Converts context-dependent phrases (\"What came after that?\", \"Tell me more about his early life\") into explicit questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the correct pages without relying on conversation context. Only invoked when conversation history is non-empty. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### SummarizeAgent\n\nCompresses the oldest conversation turns into a single `[Session summary: …]` assistant turn when a session exceeds `conversation_history_turns`. Prevents unbounded context growth across long sessions. Emits a `notice` SSE event the first time compression occurs so the user can see that earlier context was condensed. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### ContextAgent\n\nBuilds token-budgeted context packs for MCP tool consumers. See [Section 20 — Context Packs](#20-context-packs) for full detail.\n\n### SearchDecomposeAgent\n\nDecomposes a web search intent into 1–4 terse keyword search strings optimised for authoritative source retrieval. Uses a separate prompt from `QueryAgent`'s question decomposition — search decomposition asks \"what distinct search strings would find the best sources?\" (terse keyword phrases) rather than \"what distinct questions does this ask?\" (natural-language sub-questions). See [Web search fan-out](#web-search-fan-out) in the IngestAgent section.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n| `session` | `.jsonl` | `claude session`, `codex session`, `cursor session`, `ai session`, `session history` | Extracts human-readable turns from AI coding session transcripts. Supports Claude Code JSONL format and Codex/Cursor format. Filters tool calls, thinking blocks, and sub-agent scaffolding. Short turns (< 20 assistant words, < 3 user words) are skipped. No external dependencies. |\n\n### Session Skill — AI Session History Ingestion\n\nThe `session` skill turns AI coding session history files (`.jsonl`) into searchable wiki pages capturing the problem-solving knowledge exchanged during the session.\n\n**Supported formats**\n\n| Format | File origin | Detection |\n|--------|-------------|-----------|\n| Claude Code | `~/.claude/projects//.jsonl` | `obj.type` ∈ `{\"user\", \"assistant\"}` with nested `obj.message` |\n| Codex / Cursor | Export from OpenAI Codex or Cursor IDE | `obj.role` at top level, no `message` wrapper |\n\nFormat is auto-detected from the first 30 parseable lines. Files that match neither format are parsed as Codex (fallback).\n\n**Filtering rules**\n\n| Content | Kept? | Reason |\n|---------|-------|--------|\n| User text ≥ 3 words | ✓ | Substantive question or instruction |\n| Assistant text ≥ 20 words | ✓ | Substantive answer |\n| User text < 3 words (`\"ok\"`, `\"yes\"`) | ✗ | Too terse to be useful |\n| Assistant text < 20 words | ✗ | Acknowledgement or one-liner |\n| `isSidechain: true` lines | ✗ | Sub-agent scaffolding (not the user's conversation) |\n| `tool_use` blocks | ✗ | Shell commands / file writes — not final output |\n| `tool_result` blocks | ✗ | Raw output — avoids leaking file contents or credentials |\n| `thinking` blocks | ✗ | Internal reasoning — not the final answer |\n| `permission-mode`, `system`, `last-prompt` lines | ✗ | Session metadata, not conversation |\n\nAfter extraction the text passes through Synthadoc's standard pre-LLM source sanitizer\n(zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars,\ninstruction-override phrases) — the same step applied to every PDF, DOCX, URL, and other source type.\nSee [§29 Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer).\n\n**Output format**\n\nEach kept turn is labelled and separated by `---`:\n\n```\n[USER]\nHow do I implement a sliding window algorithm in Python?\n\n---\n\n[ASSISTANT]\nA sliding window algorithm maintains a contiguous subarray by advancing two\npointers simultaneously…\n```\n\n**`suggested_slug`**\n\nThe skill sets `metadata[\"suggested_slug\"]` to `session-YYYY-MM-DD-` where the date comes from the file's `mtime` and the topic from the first substantive user turn (first 6 words, slugified). Example: `session-2026-07-15-how-do-i-implement-sliding`.\n\n**Limitations**\n\n- No chunking in v1.1 — very long sessions are truncated at `max_source_chars` (default 400 000 chars). Split large archives into individual `.jsonl` files.\n- Tool output is excluded by design — re-ingest the original source files if you need the file contents in the wiki.\n- Re-ingesting the same `.jsonl` file uses Synthadoc's standard source-hash dedup — no duplicate pages are created.\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n**Compound identifier expansion:** When a token contains underscores, the tokenizer emits both the compound form and each component as separate tokens (`capex_growth` → `[\"capex_growth\", \"capex\", \"growth\"]`). This expansion is applied at both index time and query time so that human-friendly queries match programmatic identifiers and vice versa. Leading/trailing underscores are stripped before splitting; no empty strings are emitted.\n\n**TF fallback for small corpora:** When all BM25 scores in a result set are zero or negative (IDF collapse on a corpus of ≤ 2 documents, or a term present in every page), the search system falls back to normalised term frequency (TF) scoring: `TF(doc, query) = Σcount(term, doc) / len(doc_tokens)`. Results are returned in the same format as BM25 results with a `tf_fallback: True` flag on each `SearchResult`. This field is recorded in the audit trail and consumed by gap detection (see below).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp, cascade_links_removed_from: [str]}` _(cascade field added v1.0.2)_ |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n| `GET` | `/graph` _(v1.0.0)_ | — | `{status, node_count, edge_count, cluster_count, nodes: [...], edges: [...]}` or `{status: \"computing\"}` on first call |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n**External file paths** (outside the wiki root) are supported when the request comes from\n`127.0.0.1` or `::1` and the payload includes `\"allow_external_paths\": true`. The CLI sets\nthis flag automatically for all local file sources. Remote clients cannot set this flag\n(the server ignores it for non-localhost connections) to prevent arbitrary file reads\nin server-exposed deployments.\n\nThis is required for ingesting files that live outside the wiki directory by design —\nfor example, Claude Code session transcripts at `~/.claude/projects//.jsonl`.\n\n**Client behaviour summary:**\n\n| Client | Sends `allow_external_paths` | Can ingest outside wiki root? |\n|--------|------------------------------|-------------------------------|\n| CLI (`synthadoc ingest `) | `true` (automatic, local file paths only) | Yes — server is localhost |\n| Obsidian plugin | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Web UI | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Direct HTTP API (localhost) | Set manually in request body | Yes — if request from 127.0.0.1/::1 |\n| Direct HTTP API (remote) | Ignored by server | No — server silently treats as `false` |\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 1.1.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n**Reading View default:** `synthadoc plugin install` and `synthadoc plugin upgrade` write `\"defaultViewMode\": \"preview\"` to `.obsidian/app.json` after copying the plugin files. This causes Obsidian to open new notes in Reading View by default, which is required for `^[filename:L-L]` citation chips to render. Only `defaultViewMode` is written; all other keys in `app.json` are preserved. If `app.json` is malformed JSON the write is skipped and a warning is logged — the install still succeeds.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports two things: dangling slugs (pages listed in ROUTING.md that no longer exist in the wiki) and unassigned slugs (pages that exist in index.md but are not listed in any ROUTING.md branch). Enabled only when ROUTING.md exists. **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n| `Graph: show knowledge graph` _(v1.1.0)_ | Open the knowledge graph panel — draggable modal with Canvas force-directed graph, type filter dropdown, cluster legend, hover tooltip showing title/slug/type/state/cluster/connections, and click-to-open page in current pane. System pages (index, overview, dashboard, purpose, log) are excluded. Capped at 300 most-connected nodes for large wikis with an explanatory banner. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [-w wiki] — generate ROUTING.md from index.md branch structure\n│ ├── validate [-w wiki] — report dangling slugs and cross-branch duplicates\n│ └── clean [-w wiki] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [-w wiki]\n├── candidates\n│ ├── list [-w wiki]\n│ ├── promote |--all [-w wiki]\n│ └── discard |--all [-w wiki]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** subscription, or want to use the free **Opencode Zen** tier, you can use either as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode Zen (free, no API key — requires connecting first: run `opencode` → `/connect` → select Zen):\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"opencode/big-pickle\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 30\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n# Citation pass (Pass 4) tuning — these two settings work together:\n# citation_source_lines — how many lines of the source the LLM sees when placing ^[...] markers.\n# Increase if lint reports out_of_range on long sources (transcripts, PDFs).\n# citation_max_tokens — output token budget for the annotated section returned by the LLM.\n# Increase if you raise citation_source_lines and have long wiki sections.\n# citation_source_lines = 400\n# citation_max_tokens = 8192\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax`, `qwen` (DashScope), and `deepseek` providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `30` | Exponential backoff base; delays are `min(base × 2^(attempt−1), 300)` seconds |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Emit `logger.warning` in server log when per-job ingest cost exceeds this threshold; job continues normally |\n| `cost.hard_gate_usd` | float | `2.00` | Permanently fail the ingest job (DEAD) and record a `cost_gate_exceeded` audit event when per-job cost exceeds this threshold |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × (backup_count + 1)` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n| `ingest.max_source_chars` | int | `32000` | Character limit applied to each source before the LLM call. Sources exceeding this limit are truncated; the page's `sources:` frontmatter entry gets `truncated: true` and lint emits a warning. Override per-run with `--max-source-chars N`. _(v1.0.0)_ |\n| `ingest.citation_source_lines` | int | `400` | Number of source lines the LLM sees during Pass 4 (citation annotation). Increase if lint reports `out_of_range` on long sources such as transcripts or PDFs. Raise `citation_max_tokens` in proportion when increasing this value. |\n| `ingest.citation_max_tokens` | int | `8192` | Output token budget for the annotated section returned by Pass 4. Increase when `citation_source_lines` is raised or wiki sections are long, to avoid truncated annotations. |\n| `query.context_wiki_pct` | float | `0.60` | Fraction of the model context window reserved for wiki source pages. _(v1.0.0)_ |\n| `query.context_history_pct` | float | `0.20` | Fraction reserved for conversation history. _(v1.0.0)_ |\n| `query.context_system_pct` | float | `0.15` | Fraction reserved for system prompt and instructions. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_index_pct` | float | `0.05` | Fraction reserved for the wiki index preamble. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_window` | int | _(auto)_ | Override the built-in context window lookup for the configured model. Use when running a local or custom model whose window size is not in the built-in table. _(v1.0.0)_ |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement is wired into the ingest pipeline: `Orchestrator._guard_ingest_cost()` is called after each ingest LLM call with the returned cost, before the job is marked complete.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | `logger.warning` in server log; job continues normally |\n| `hard_gate_usd` | $2.00 | Server/ingest path: records `cost_gate_exceeded` audit event + permanently fails job (DEAD). Interactive CLI: prompts `Proceed? [y/N]` |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `min(backoff_base_seconds × 2^(attempt − 1), 300)` \nWith the default base of 30 s: attempt 1 waits 30 s, attempt 2 waits 60 s, attempt 3 waits 120 s (all capped at 300 s). The `retry_after` timestamp is stored in the jobs table; `dequeue()` skips any job whose `retry_after` is still in the future. Applied only to transient errors (network timeouts, connection failures, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` by default. The bind address is configurable via `server.host` in `config.toml` (e.g. `host = \"0.0.0.0\"` to expose on all interfaces for LAN access). No built-in authentication — restrict via firewall when exposing beyond loopback. Remote access without a reverse proxy is not recommended on shared or networked machines.\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs and unassigned slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `-w ` / `--wiki ` to target a specific wiki.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; when all local source files are missing from disk the page is transitioned to `archived` (if only some sources are missing it is transitioned to `stale` instead). Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: lint evaluates all local (non-URL) source paths listed in the page's frontmatter. If **all** of them are missing from disk, the page is transitioned to `archived`. If only **some** are missing — meaning valid content from at least one source still exists — the page is transitioned to `stale` instead, preserving it for re-ingest rather than discarding it. _(v1.0.2: previously any single missing source triggered an immediate archive.)_\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Cascade link cleanup\n\nWhenever a page transitions to `archived` — whether triggered manually (`synthadoc lifecycle archive`), via the Obsidian Lifecycle modal, the MCP `synthadoc_lifecycle` tool, or automatically by lint detecting a missing source — Synthadoc immediately scans all other active pages and removes every `[[archived-slug]]` wikilink pointing to the now-archived page.\n\n- **Inline links** — `[[archived-slug]]` becomes plain text (the display text if an alias was used, otherwise the slug itself).\n- **List-item links** — a list item whose only content is `[[archived-slug]]` is dropped entirely.\n- **Archived pages are skipped** — pages already archived are not rewritten.\n- **System pages are skipped** — index, overview, dashboard, log, and purpose pages are excluded.\n\nThe cleanup runs synchronously before the response is returned, so the wiki is consistent immediately. The CLI and MCP responses include a `cascade_links_removed_from` field listing the affected slugs. During `lint run`, cascade is batched once after all auto-archive transitions complete, and the count is added to `dangling_links_removed` in the lint report.\n\nLint's existing periodic dangling-link cleanup (`lint run --scope orphans`) remains in place as a safety net for any dead links that predate this feature or arrive through other paths.\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [...], \"input_tokens\": 1240, \"output_tokens\": 380, \"tokens_used\": 1620}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […], \"input_tokens\": N, \"output_tokens\": N, \"tokens_used\": N}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Streaming Token Audit\n\nStreaming LLM responses do not return token counts in the same way as blocking calls. To track real usage in the audit trail:\n\n**Provider-side collection** — each `LLMProvider` subclass exposes two instance attributes: `last_stream_input_tokens` and `last_stream_output_tokens`. These default to `0` and are populated from the API-specific usage field when the stream ends:\n\n| Provider | Mechanism | Verified |\n|---|---|---|\n| OpenAI | `stream_options={\"include_usage\": True}` passed to `chat.completions.create()`; final chunk carries `chunk.usage.prompt_tokens` / `completion_tokens` with empty `choices` | ✅ |\n| Anthropic | `message_start` event → `event.message.usage.input_tokens`; `message_delta` event → `event.usage.output_tokens` | ✅ |\n| Ollama | Final chunk with `done=True` → `prompt_eval_count` / `eval_count` | ✅ |\n| DeepSeek | Same `OpenAIProvider` path as OpenAI; DeepSeek's OpenAI-compatible API supports `stream_options` | ✅ |\n| MiniMax (reasoning: M2.5+) | Detects `` in stream → falls back to blocking `complete()` → captures exact counts from `resp.usage` | ✅ |\n| MiniMax (non-reasoning, e.g. M3 thinking=disabled) | Same `OpenAIProvider` path; MiniMax API silently ignores `stream_options`. Falls back to character-based estimate (÷ 3.5 chars/token) from prompt + answer lengths. Accuracy ±20%. | ✅ estimated |\n| Gemini | Same `OpenAIProvider` path via `generativelanguage.googleapis.com/v1beta/openai/`; Google's compatibility layer honours `stream_options` — verified live (Gemini 2.5 Flash Lite, 50K tokens) | ✅ |\n| Groq | Same `OpenAIProvider` path; Groq's OpenAI-compatible API supports `stream_options` | ✅ |\n| Qwen (DashScope) | Same `OpenAIProvider` path via DashScope; `stream_options` honoured — verified live (qwen-plus, 28K tokens) | ✅ |\n| Qwen (Ollama) | Ollama path → `prompt_eval_count` / `eval_count` | ✅ |\n\n**Forwarding to `done` event** — after `complete_stream()` is exhausted, `QueryAgent.run_stream()` reads the provider attributes and includes them in the final `done` event payload: `input_tokens`, `output_tokens`, `tokens_used`.\n\n**Audit record** — `Orchestrator.query_stream()` consumes the `done` event and calls `estimate_cost(model, input_tokens, output_tokens, is_local)` from `synthadoc/providers/pricing.py`, then calls `AuditDB.record_query()` with the real token count and computed cost. The audit row at `GET /audit/queries` will therefore always show non-zero values for cloud providers.\n\nWhen a provider does not report streaming usage (e.g. a future provider not yet wired), the attributes stay at `0`; the audit row records `tokens=0, cost_usd=0.0` rather than crashing, matching the pre-fix behaviour.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 5) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 5 # turns to include in each request (default: 5; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp, cascade_links_removed_from: [str]}` or `{error, cascade_links_removed_from: []}` _(cascade field added v1.0.2)_ | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions for Codex CLI / OpenCode (if present)\n├── CLAUDE.md ← same content; loaded by Claude Code (if present)\n├── GEMINI.md ← same content; loaded by Gemini CLI (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── hooks/ ← user hook scripts (always included)\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ ├── extracted/ ← text sidecars + PDF pagemaps (always included)\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `logs/`.\n\n> **v0.9.3 fix:** `hooks/` and `.synthadoc/extracted/` were previously missing from backups. Omitting `extracted/` caused the Source Viewer and Provenance citation modal to show \"Could not read source file\" after restore.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## 29. Pre-LLM Source Sanitizer\n\nEvery source document passes through a sanitizer immediately after text extraction, before any LLM call. The sanitizer removes six categories of content that could manipulate the LLM or degrade compilation quality:\n\n| Category | Action | Warning logged? |\n|---|---|---|\n| Zero-width characters (U+200B, U+200C, U+200D, U+FEFF) | Removed silently | No |\n| Bidi override characters (U+202A–U+202E, U+2066–U+2069) | Removed | Yes |\n| HTML comments (``) | Removed silently | No |\n| Hidden CSS spans (`display:none`, `visibility:hidden`) | Removed silently | No |\n| Base64 blobs ≥ 200 consecutive characters | Replaced with `[base64 content removed]` | Yes |\n| Instruction-override phrases (8 patterns: \"ignore previous instructions\", \"disregard the above\", \"override your system prompt\", etc.) | Replaced with `[redacted]` | Always |\n\nWhen content is removed and a warning is appropriate, the server logs a `WARN` entry:\n```\n[WARN] sanitizer stripped content from 'papers/survey.pdf': bidi overrides, instruction-override phrase\n```\n\nThis is a pure pre-processing step with zero LLM cost. Truncation (if configured) is applied after sanitization, so the character budget is consumed by clean text only.\n\n---\n\n## 30. Per-Source Truncation Flag\n\nLarge source documents are truncated before they reach the LLM. By default the limit is **32,000 characters** (~8,000 tokens), raised from the prior hardcoded 8,000-character limit. When a source exceeds this limit, its compiled wiki page records a `truncated: true` flag in its `sources:` frontmatter entry.\n\n### Configuration\n\n```toml\n[ingest]\nmax_source_chars = 32000 # default; raise for large PDFs or books\n```\n\n### Per-ingest override\n\nThe limit can be raised for a single ingest run without modifying the config:\n\n```bash\nsynthadoc ingest papers/large-textbook.pdf --max-source-chars 128000\n```\n\nThe same `max_source_chars` field is accepted by the HTTP `POST /jobs/ingest` body and the MCP `synthadoc_ingest` tool.\n\n### Lint surfacing\n\n`synthadoc lint` emits a warning for any page that has a truncated source:\n\n```\n[WARN] quantum-computing.md: source 'papers/large-survey.pdf' was truncated at ingest\n (source exceeded max_source_chars=32000 — 87,412 chars in source).\n To re-ingest with a higher limit (this source only):\n synthadoc ingest papers/large-survey.pdf --max-source-chars 128000\n To raise the limit for all future ingests:\n set [ingest] max_source_chars = 128000 in your config\n```\n\n---\n\n## 31. Proportional Context Budget\n\nQuery context is allocated proportionally to the configured model's context window, replacing the prior hard cap. The default allocation:\n\n| Slice | Default | Purpose |\n|---|---|---|\n| `context_wiki_pct` | 60% | Wiki page content (ranked by BM25 + vector score) |\n| `context_history_pct` | 20% | Chat turn history (newest-first, oldest dropped when over budget) |\n| `context_system_pct` | 15% | System prompt and purpose document |\n| `context_index_pct` | 5% | `ROUTING.md` and search index |\n\nPercentages must sum to ≤ 100. Configure in `[query]`:\n\n```toml\n[query]\ncontext_window = 0 # 0 = auto-detect from known model map\ncontext_wiki_pct = 60\ncontext_history_pct = 20\ncontext_system_pct = 15\ncontext_index_pct = 5\n```\n\n> **Note:** `context_system_pct` and `context_index_pct` are parsed and validated but not yet enforced as hard caps in this release — the system prompt uses a conservative fixed limit, which is well within the configured 15% slice for all supported models. Full per-slice enforcement is planned for v1.1.\n\n### Model context window map\n\nSynthadoc ships a built-in prefix-matched table:\n\n| Model | Context window |\n|---|---|\n| `claude-opus-4*`, `claude-sonnet-4*`, `claude-haiku-4*` | 200,000 tokens |\n| `gpt-4o*`, `gpt-4-turbo*` | 128,000 tokens |\n| `gpt-4` (exact) | 8,192 tokens |\n| `gpt-3.5-turbo*` | 16,385 tokens |\n| Unknown / fallback | 128,000 tokens |\n\nSet `context_window = N` in config to override the map (useful for local Ollama models with non-standard context sizes).\n\nAt the defaults on a 200k-token model: wiki slice ≈ 480,000 chars — far more than any practical query needs. On constrained models, the budget scales down automatically so the same config works across model families.\n\n---\n\n## 32. Knowledge Graph\n\nSynthadoc computes a wikilink graph across all active and draft pages during every lint run. The graph is stored in two tables in `audit.db`:\n\n```sql\nCREATE TABLE graph_nodes (\n slug TEXT PRIMARY KEY,\n cluster_id INTEGER NOT NULL,\n updated_at TEXT NOT NULL\n);\n\nCREATE TABLE graph_edges (\n from_slug TEXT NOT NULL,\n to_slug TEXT NOT NULL,\n weight INTEGER NOT NULL DEFAULT 1,\n edge_type TEXT NOT NULL DEFAULT 'mixed',\n PRIMARY KEY (from_slug, to_slug)\n);\n```\n\nEdge weight is a composite signal from two sources:\n\n| Edge type | Source | Weight contribution |\n|-----------|--------|---------------------|\n| `wikilink` | `[[slug]]` occurrences in page body | +1 per occurrence |\n| `co_source` | pages compiled from the same source file (matched by hash) | +2 per shared source |\n\nMost edges are `mixed` (have both types). Pure `co_source` edges surface hidden relationships — pages compiled from the same source document are linked immediately after ingest, before any wikilinks are created. `edge_type` is stored on each edge so the web UI can render co-source edges with a different visual style.\n\nCluster IDs are assigned by Louvain community detection — pages that link densely to each other end up in the same cluster.\n\n### REST API\n\n**`GET /graph`** — returns the full graph with per-node enrichment:\n\n```json\n{\n \"status\": \"ready\",\n \"node_count\": 45,\n \"edge_count\": 123,\n \"cluster_count\": 4,\n \"nodes\": [\n {\"slug\": \"quantum-error-correction\", \"title\": \"Quantum Error Correction\",\n \"type\": \"concept\", \"state\": \"active\", \"cluster_id\": 2}\n ],\n \"edges\": [\n {\"from\": \"quantum-error-correction\", \"to\": \"shor-algorithm\", \"weight\": 3, \"edge_type\": \"mixed\"}\n ]\n}\n```\n\nWhen the graph tables are empty (first call after upgrade, or before the first lint run), the server returns `{\"status\": \"computing\"}` and starts a background build immediately. The web UI polls every 2 seconds and renders the graph when the status becomes `\"ready\"`.\n\n### Web UI\n\nThe **Graph** tab (alongside Chat in the top nav) renders a D3.js force-directed graph. Nodes are colored by cluster. Edge thickness is proportional to weight (1–4 px). Pure `co_source` edges (no wikilink) are rendered with a dashed stroke, making hidden source-based relationships visually distinct from explicit cross-links. Clicking a node opens a sidebar showing the page title, type and lifecycle state badges, cluster assignment, and an **\"Ask about this →\"** button that switches to the Chat tab with a pre-filled query.\n\nControls: zoom in/out (scroll or pinch), drag nodes, filter by page type.\n\n### System page exclusion _(v1.1.0)_\n\nThe five structural system pages — `overview`, `index`, `dashboard`, `purpose`, and `log` — are excluded from both `graph_nodes` and `graph_edges`. These pages link to nearly every other page and would act as false hubs, inflating edge counts and distorting Louvain cluster detection. System page exclusion is applied inside `_build_graph()` before any edge computation.\n\n### Obsidian knowledge graph panel _(v1.1.0)_\n\n→ Full description: [§33 Multi-Platform Agent Skill Files](#33-multi-platform-agent-skill-files) covers scaffold; the graph panel itself is documented in [§8 Obsidian Plugin](#8-obsidian-plugin) command table.\n\nThe Obsidian plugin includes a **draggable modal** knowledge graph panel (`Graph: show knowledge graph` command). It fetches from the same `GET /graph` endpoint as the web UI and renders a force-directed Canvas 2D simulation using Verlet integration (no external library). System pages are already excluded from the server response.\n\n| Feature | Obsidian panel | Web UI Graph tab |\n|---------|----------------|-----------------|\n| Engine | Canvas 2D + Verlet | D3.js SVG |\n| Node click | Opens the wiki page in the current pane | Opens the node sidebar with \"Ask about this →\" |\n| **Best for** | **Navigating to and editing pages** | **Asking questions about a page** |\n| Draggable panel | Yes — drag header to reposition | Fixed tab layout |\n| Node cap | 300 most-connected (see note) | No cap |\n| Cluster legend | Yes | Yes |\n| Type filter | Yes | Yes |\n\n**Rule of thumb:** Use the **web UI** when you want to ask questions about a node. Use the **Obsidian panel** when you want to navigate to and edit a page.\n\n**Node cap rationale:** The Obsidian panel uses a hand-written Verlet integrator with O(n²) pairwise charge repulsion — every node repels every other node on each animation tick. At 300 nodes that is ~90,000 operations per frame, which runs smoothly at 60 fps inside Obsidian's Electron renderer. At 500 nodes it is ~250,000 operations per frame and starts to lag noticeably. The web UI has no cap because D3's `forceManyBody` uses a Barnes-Hut tree approximation (O(n log n)), which scales to thousands of nodes without frame-rate issues. When the cap is hit, nodes are ranked by degree and the 300 most-connected are shown; a banner reports the total page count and the number of hidden nodes, and suggests using the type filter to bring the set below 300.\n\n→ User walkthrough: [Quick-Start Guide §24 — Knowledge Graph](docs/user-quick-start-guide.md#knowledge-graph)\n\n---\n\n## 33. Multi-Platform Agent Skill Files\n\n\n\n`synthadoc init` and `synthadoc scaffold` write three agent skill files at the wiki root:\n\n| File | Format | Read by |\n|------|--------|---------|\n| `AGENTS.md` | OpenAI Agents SDK / Codex / OpenCode | Codex CLI, OpenCode, generic agents |\n| `CLAUDE.md` | Claude Code / Anthropic | Claude Code (loaded automatically when the wiki folder is opened) |\n| `GEMINI.md` | Gemini CLI | Gemini CLI |\n\nAll three files carry **identical content**: a title block, the LLM-generated domain guidelines for the wiki, a quick-reference CLI command table, and an MCP tool table. The only difference is the heading comment. This means any AI coding tool that opens the wiki root gets full Synthadoc guidance without manual setup.\n\n### Content generated by `scaffold`\n\nEach skill file is generated from the same LLM call that produces `purpose.md` and `index.md`. The LLM receives the current wiki page list and domain description, then generates:\n\n1. **Domain label** — a precise human-readable name inferred from wiki content (e.g. `History of Computing`, not `My Wiki`)\n2. **Domain guidelines** — 3 domain-specific ingest and query guidelines as plain sentences\n3. **Quick Reference table** — pre-filled with the current wiki name and port\n\nIf the wiki was created before v1.1.0, running `synthadoc scaffold` once regenerates all three files and adds `CLAUDE.md` and `GEMINI.md` to any wiki that only had `AGENTS.md`.\n\n### Template source\n\nThe instruction body is defined in `synthadoc/cli/_init.py` as `_AGENT_INSTRUCTION_BODY` and shared by all three templates, keeping the files permanently in sync. The scaffold agent substitutes `{domain}`, `{guidelines}`, and `{port}` at write time.\n\n---\n\n## 34. Synthadoc Scaffold Marker\n\n\n\nThe scaffold marker `` (written as an HTML comment) separates **user-authored content** (preserved) from **scaffold-managed content** (rewritten on each `synthadoc scaffold` run).\n\n### Single-marker mode (index.md)\n\nThe marker appears once, below the H1 heading in `index.md`. Everything above the marker is the user zone; everything below is replaced with new LLM-generated categories.\n\n```markdown\nMy custom introduction — maintained by hand.\n\n\n\n## Pioneers and Visionaries\n- [[alan-turing]]\n...\n```\n\nIf the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### Per-section multi-marker mode (purpose.md)\n\n`purpose.md` uses a **per-section** approach: the marker appears inside each `## Section` heading. Within a section, content the user writes **above** the marker is preserved; content below is replaced with fresh LLM output. Sections without a marker in the existing file are replaced entirely (and a marker is added). Sections in the existing file that the scaffold template does not include are kept unchanged.\n\n```markdown\n## Overview\n\nMy custom overview note.\n\n\n\nLLM-generated overview content goes here.\n\n## What Belongs\n\n\n\nLLM-generated scope definition.\n```\n\nThis design lets users annotate any section of `purpose.md` without losing their notes on the next scaffold run.\n\n### Marker preservation guarantee\n\n- `preserve_user_zone()` in `scaffold_agent.py` handles both modes.\n- If a section has no marker in the existing file, `preserve_user_zone()` treats the entire section body as user content and preserves it, replacing only what falls below any marker it finds.\n- The scaffold NEVER touches `config.toml`, `dashboard.md`, or any wiki page (`wiki/*.md`).\n\n→ User walkthrough: [Quick-Start Guide §14 — Enrich the wiki with scaffold](docs/user-quick-start-guide.md#scaffold)\n\n---\n\n## Customization\n\nSynthadoc exposes four extension points — all hot-loaded, no server restart required.\n\n### Custom skills (new file formats)\n\nSubclass `BaseSkill` (Apache-2.0 — no AGPL obligation on your skill code), drop the folder in `/skills/` or `~/.synthadoc/skills/`, and Synthadoc picks it up on the next ingest. Skills match by file extension or intent prefix and support any Unicode text, including CJK prefixes.\n\n→ Full interface, manifest format, and examples: [§5 Skills System](#5-skills-system) and [§17 Plugin Development Guide](#17-plugin-development-guide)\n\n### Custom LLM providers\n\nSubclass `LLMProvider` from `synthadoc/providers/base.py` (Apache-2.0) and place the file in `~/.synthadoc/providers/` or the wiki `providers/` directory. Switch active provider with one config line.\n\n→ Full interface and wiring instructions: [§10 Configuration — Provider switching](#10-configuration)\n\n### Hooks\n\nShell scripts (any language) that fire on `on_ingest_complete` and `on_lint_complete`. Receive a JSON context on stdin. Set `blocking = true` to gate the operation on the hook's exit code — useful for CI pipelines and quality gates.\n\n→ Full event schema, context JSON, and blocking behaviour: [§11 Hook System](#11-hook-system)\n\n### Cache control\n\nThree cache layers (embedding, LLM response, provider prompt cache) invalidate automatically on source-file change (SHA-256). Force a fresh call with `--force`, or wipe all cached responses with `synthadoc cache clear -w `.\n\n→ Layer-by-layer breakdown and invalidation rules: [§12 Cache System](#12-cache-system)\n\n### Multi-Platform Agent Skill Files\n\nEvery Synthadoc wiki ships three companion files that give AI coding agents a complete, self-contained operating guide for the wiki:\n\n| File | Platform |\n|---|---|\n| `AGENTS.md` | Codex CLI, OpenCode, and any agent that reads the AGENTS.md convention |\n| `CLAUDE.md` | Claude Code (this session's tool) — highest-priority instruction source in Claude's hierarchy |\n| `GEMINI.md` | Gemini CLI — relevant because Synthadoc's default LLM provider is `gemini-2.5-flash-lite` |\n\nAll three files share identical body content generated from the same template; they differ only in their H1 heading so each agent recognises its own file first. The body covers:\n\n- **Domain guidelines** — LLM-generated bullet list of domain-specific ingest and query rules (produced by `scaffold`)\n- **Quick reference table** — every common CLI command with the correct `synthadoc` syntax\n- **Server startup** — how to start the server and verify it is running\n- **Ingest examples** — local file, URL, YouTube, agent session `.jsonl`, force reingest, dry-run\n- **Query** — basic and streaming query syntax; citation marker format\n- **Lint** — when to run and what each check does\n- **Lifecycle** — transition commands with state names\n- **Page schema** — frontmatter structure with all fields\n- **MCP tools table** — all 12 tools with purpose descriptions\n\n**How the files are created and updated:**\n\n1. `synthadoc init` — writes all three files with default domain guidelines and the wiki's configured port.\n2. `synthadoc scaffold` — regenerates all three with LLM-produced domain guidelines derived from the current wiki state. The `scaffold` command prints a confirmation line for each file.\n\n**Use cases:**\n\n- Open Claude Code in a Synthadoc wiki directory and it immediately knows how to ingest, query, lint, and manage lifecycle — no additional setup. Claude Code can also start the server in background mode (`synthadoc serve -w -b`), stop it cleanly, and orchestrate multi-step workflows, effectively acting as a control plane for the wiki.\n- Gemini CLI reads `GEMINI.md` and can run `synthadoc` commands with the correct flags without asking the user for help.\n- CI pipelines with Codex CLI use `AGENTS.md` to drive automated lint and lifecycle transitions on PR merge.\n- Agent session files produced by Claude Code (`.jsonl` under `~/.claude/projects/`) can themselves be ingested into the wiki, capturing the reasoning behind design decisions alongside the source documents.\n\n→ See [README.md — Interfaces & Integration](../README.md#interfaces--integration) for the feature comparison row and quick-start instructions.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v1.1.0\n\n- **Weighted knowledge graph edges** — graph edges now carry two signals: wikilink occurrences (+1 each) and co-source connections — pages compiled from the same source file (matched by SHA-256 hash) — (+2 per shared source). `edge_type` field added to `graph_edges` table (`wikilink`, `co_source`, or `mixed`). `GET /graph` exposes `edge_type` per edge. Web UI renders edge thickness proportional to weight (1–4 px) and dashed lines for pure co-source edges. Co-source edges appear immediately after ingest — before any wikilinks — surfacing hidden relationships in the graph. Schema migrated to version 3.\n- **Multi-platform agent skill files** — `synthadoc init` and `synthadoc scaffold` write `CLAUDE.md`, `AGENTS.md`, and `GEMINI.md` at wiki root; all three carry LLM-generated domain guidelines and the full CLI quick-reference so Claude Code, Codex CLI, OpenCode, Pi, and Gemini CLI users get first-class guidance without manual setup. Run `synthadoc scaffold` once on any pre-v1.1.0 wiki to generate the new files. See [§33](#33-multi-platform-agent-skill-files).\n- **Session history ingestion skill** — `.jsonl` session files from Claude Code and Codex CLI are ingested as wiki pages; human turns and substantive assistant responses extracted; tool calls, thinking blocks, and sub-agent scaffolding skipped; format auto-detected; `suggested_slug` derived from session date and first user message. Converts ephemeral problem-solving knowledge into searchable wiki entries. See [§5 Skills System — Session Skill](#session-skill--ai-session-history-ingestion).\n- **Obsidian knowledge graph panel** — new `Graph: show knowledge graph` command in the Obsidian plugin opens a draggable modal with a Canvas 2D force-directed graph. Features: Verlet integration, type filter dropdown, cluster legend, hover tooltip (title, slug, type, state, cluster, connections), click-to-open page in current pane, node drag to repin. Closes only via the × button — Escape and outside-click do not close the panel, matching the behaviour of other Synthadoc modals. **Large-wiki cap:** when the node set exceeds 300 (after any type filter is applied), nodes are ranked by degree (total edge count) and the 300 most-connected are displayed; edges to capped-out nodes are removed; a banner explains the cap and suggests using the type filter to narrow the view below 300. See [§8 Obsidian Plugin](#8-obsidian-plugin) and [§32 Knowledge Graph — Obsidian panel](#obsidian-knowledge-graph-panel-v110).\n- **Per-section scaffold markers in `purpose.md`** — `purpose.md` now uses per-section `` markers. Users can write content above the marker in any `## Section` block; that content is preserved on every re-scaffold. A single-marker mode for `index.md` continues to work as before. See [§34 Synthadoc Scaffold Marker](#34-synthadoc-scaffold-marker).\n- **System pages excluded from knowledge graph** — `overview`, `index`, `dashboard`, `purpose`, and `log` are filtered out of `_build_graph()` before any edge computation so they cannot act as false hubs in Louvain cluster detection. Both the Obsidian graph panel and the web UI Graph tab reflect this exclusion.\n- **Web UI graph cluster legend** — the web UI Graph tab now shows a cluster legend below the canvas (matching the Obsidian plugin panel), with a color swatch and label for each Louvain cluster present in the filtered node set.\n- **`_merge_section_into_page` deduplication** — force re-ingest of the same source file (e.g. an updated `.jsonl` session) now replaces the page body rather than appending a duplicate section. `_files_match()` normalizes absolute vs. relative paths so the same file is recognized regardless of how it was passed to the ingest command.\n- **`purpose.md` domain label inference** — `scaffold` now infers a precise `domain_label` from current wiki content (e.g. `History of Computing`) instead of using a generic placeholder; the label is used in all three agent skill files and in `purpose.md`'s Overview section.\n- **CI: Windows runner fix** — `scripts/sync_plugin.py` and `scripts/sync_web_ui.py` now use `shutil.copy2` instead of `cp -f`, resolving CI failures on Windows runners for the sync check.\n\n### v1.0.2\n\n- **Cascade link cleanup on archive** — archiving any page immediately removes every `[[slug]]` wikilink pointing to it from all other active pages, without waiting for the next `lint run`. Inline links become plain text; list-item-only links are dropped entirely. The HTTP `POST /lifecycle/transition` and MCP `synthadoc_lifecycle` responses now include a `cascade_links_removed_from` field listing the rewritten page slugs. During `lint run`, cascade runs once after all auto-archive transitions complete and the count is included in the lint report's `dangling_links_removed` field. System pages (index, dashboard, purpose, log, overview) and already-archived pages are skipped. The existing dangling-link lint pass remains as a safety net for dead links that predate this release.\n- **Multi-source partial-archive fix** — when a page has multiple local source files and only some are missing from disk, lint now transitions the page to `stale` (not `archived`), preserving it for re-ingest. The page is only archived when every local source file is gone. URL sources are not affected by this check.\n\n### v1.0.1\n\n- **Scaffold JSON extraction hardened** — replaced the greedy regex used to extract the LLM's JSON response with a brace-balanced parser; eliminates false-positive matches on `[[wikilink]]` syntax that previously caused `json.loads()` failures and silent scaffold corruption\n- **Generated file validation** — scaffold now validates every LLM-generated file (index, purpose, AGENTS.md) for minimum length, required frontmatter, and `[[wikilink]]` presence before writing to disk; validation failure marks the job as failed with a structured error message rather than silently writing a malformed page\n- **ROUTING.md regeneration validation** — `routing init` and scheduled routing regeneration now validate the regenerated file for minimum line count and anchor presence before applying it; prevents a corrupt or truncated ROUTING.md from entering the wiki\n- **Scaffold failure reporting** — scaffold skip and failure reasons are now surfaced in the terminal output and job error field; previously silent failures (e.g. LLM timeout, JSON parse error) are now visible without tailing the JSON log\n- **Install-time scaffold removed** — `synthadoc install` no longer runs the LLM scaffold call during installation; scaffold is a separate step (`synthadoc scaffold`) that the user runs after setting their API key, avoiding confusing API-key-not-set errors at install time\n- **AquaFlow LLM evaluation** — 15-question M&A due-diligence benchmark across five models (MiniMax-Think M3, Claude Opus 4.8, Claude Sonnet 4.6, DeepSeek-R1, Qwen Plus); full report at `docs/example/aquaflow/evaluation/report/llm-query-benchmark.md`\n\n### v1.0.0\n\n- **Pre-LLM source sanitizer** — every source document passes through a sanitizer before any LLM call; removes six categories of potentially harmful content (zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars, instruction-override phrases); a `WARN` log entry is emitted when bidi overrides or instruction-override phrases are stripped; zero LLM cost; applied before truncation so the character budget is consumed by clean text only\n- **Per-source truncation flag** — default character limit raised from 8,000 to 32,000 characters per source (~8,000 tokens); truncated sources are flagged with `truncated: true` in the page's `sources:` frontmatter; `--max-source-chars N` CLI flag and matching `POST /jobs/ingest` body field and MCP `synthadoc_ingest` parameter override the limit per run; `synthadoc lint` emits a warning for any page with a truncated source and suggests the override command; configurable via `[ingest] max_source_chars` in `config.toml`\n- **Proportional context budget** — query context is allocated proportionally to the configured model's context window; four configurable slices: `context_wiki_pct` (60%), `context_history_pct` (20%), `context_system_pct` (15%), `context_index_pct` (5%); built-in prefix-matched model context window table covers Claude 4, GPT-4o/turbo/3.5, and GPT-4; unknown models fall back to 128,000 tokens; `context_window = N` in `[query]` overrides the table for local or custom models\n- **Knowledge graph** — wikilink graph computed across all active and draft pages during every lint run; stored in `graph_nodes` and `graph_edges` tables in `audit.db`; edge weight = number of `[[wikilink]]` references between two pages; cluster IDs assigned by Louvain community detection; `GET /graph` REST endpoint returns nodes with per-node enrichment (title, type, state, cluster) and edges; first call after upgrade triggers a background build and returns `{\"status\":\"computing\"}`; web UI **Graph** tab (D3.js force-directed graph) — nodes colored by cluster, click to see page details and an \"Ask about this →\" button that opens a pre-filled chat query\n- **Citation accuracy & rendering improvements** — six correctness fixes to the ingest citation pipeline: (1) full source text passed to the decision LLM so citations can reference any line in the document; (2) `_annotate_citations` bugs A–F resolved (false-positive matches, off-by-one line ranges, duplicate markers, escaped bracket handling, multi-paragraph span errors, and source filename normalisation); (3) `_has_citations` check added — lint warns and suggests a model upgrade when a page has zero citation markers after synthesis; (4) `Key Data` section extraction hardened so numerical facts, rates, and formulas are preserved verbatim across the annotation pass; (5) `obsidianCitationsToGfm` rendering fixed in the web UI — citation superscripts now display correctly in the chat panel alongside streamed answers; (6) reading view set as the default Obsidian display mode for wiki pages\n\n### v0.9.3\n\n- **Backup now includes `hooks/` and `.synthadoc/extracted/`** — both directories were silently omitted from backups. `hooks/` contains user-customised hook scripts; `.synthadoc/extracted/` contains per-source text sidecars and PDF pagemaps written by the ingest agent. Without these, the Source Viewer and Provenance citation modal showed \"Could not read source file\" after restore.\n- **PyPI publish workflow** — `publish.yml` triggers on GitHub Release, builds the wheel with hatchling, and uploads to PyPI via OIDC trusted publishing (no API token stored). `pip install synthadoc` now works.\n- **Web UI bundled in pip package** — `synthadoc/data/web-ui/dist/` is now included in the wheel. `synthadoc web` previously returned 503 on pip-installed instances because the React dist was not packaged.\n- **CI sync checks** — PRs now fail if `synthadoc/data/obsidian-plugin/`, `synthadoc/data/web-ui/`, or `README-pypi.md` are out of sync with their source. All sync-check steps use `shell: bash` for Windows runner compatibility.\n- **`bump_version.py` extended** — now also updates `synthadoc/data/obsidian-plugin/manifest.json` alongside the source manifest, preventing version drift between the bundled copy and the plugin source.\n\n### v0.9.2\n\n- **`synthadoc/data/web-ui/dist/` packaged** — web UI dist files committed to the repo and bundled in the wheel via hatchling artifacts.\n- **`scripts/sync_web_ui.py`** — helper script to sync `web-ui/dist/` into `synthadoc/data/web-ui/dist/` after `npm run build`.\n\n### v0.9.1\n\n- **PyPI metadata** — added `readme`, `keywords`, `classifiers`, and `[project.urls]` to `pyproject.toml` for correct PyPI rendering.\n- **`README-pypi.md`** — generated from `README.md` by `scripts/prepare_pypi_readme.py`; strips `pypi-strip` blocks and rewrites relative links to absolute GitHub URLs.\n- **Branch protection compatibility** — removed post-merge `build-plugin` CI job that pushed compiled plugin directly to `main`. Plugin sync is now enforced in PRs: CI fails if `synthadoc/data/obsidian-plugin/` is out of sync.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 5). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n"}, "aaa3ab1cb0238df9a6e36c1c70345f7bf86641e1": {"t": 1785767250.334083, "n": 219658, "h": "73b0f4034c45c051", "c": "# Synthadoc — Design Document\n\n**Version:** 1.1.0 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n29. [Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer)\n30. [Per-Source Truncation Flag](#30-per-source-truncation-flag)\n31. [Proportional Context Budget](#31-proportional-context-budget)\n32. [Knowledge Graph](#32-knowledge-graph)\n33. [Multi-Platform Agent Skill Files](#33-multi-platform-agent-skill-files)\n34. [Synthadoc Scaffold Marker](#34-synthadoc-scaffold-marker)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for Codex CLI, OpenCode, and generic OpenAI agents\n CLAUDE.md ← same content as AGENTS.md; loaded automatically by Claude Code\n GEMINI.md ← same content as AGENTS.md; loaded automatically by Gemini CLI\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the full source text (bounded by `max_source_chars`) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `flag` / `skip`). Pages with `status='active'` are protected — sources that conflict with them trigger `flag` rather than `update` (RULE 1b).\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Orchestrator calls `_guard_ingest_cost()` on the returned cost: soft warn → `logger.warning` only; hard gate → `cost_gate_exceeded` audit event + `fail_permanent` (job → DEAD). On success, job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 0b — Key Data Extraction | None (regex) | Deterministic pre-processor: extracts numbers, percentages, currency amounts, ratios, formulas, underscore identifiers, and date ranges from source text via regex; appends them as a `[Key Data — extracted by pre-processor]` section before any LLM sees the text. Zero token cost; cannot hallucinate. Anchors the synthesis LLM to all quantitative candidates. |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads the full source text (bounded by `max_source_chars`) + BM25 candidates + `purpose.md` scope + status of each candidate page. Outputs per-page action: `create`, `update`, `flag`, `skip`. **RULE 1b:** a page with `status='active'` is human-reviewed and authoritative — when the source provides a conflicting value, date, formula, or conclusion, the action must be `flag`, not `update`. The only exception is a source that adds a topic the page does not yet mention at all. Decision cache version: `v2`. |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. If no citation markers are present in the returned body, a `citation_pass4_no_markers` WARNING audit event is written — the page is still saved but the condition is flagged for re-ingest with a more capable model. Zero-citation results are not cached so a subsequent re-ingest can succeed. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic. **Suppressed when TF fallback was used** (a small corpus with broad coverage may legitimately return fewer than 3 candidates — that is expected, not a gap).\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n| Citation presence (Check 5b) | Page body has ≥ 50 words and zero `^[filename:L-L]` citation markers → WARNING. Diagnostic only — does not block promotion to `active`. Indicates the annotation pass failed to produce markers, usually a model-compatibility issue (use Gemini 2.5 Flash or higher for reliable citation annotation). |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n### ActionAgent\n\nDispatches action-intent queries from the chat UI (e.g. \"activate a draft page\", \"show lint report\", \"archive a stale page\", \"run scaffold\"). Parses the user's free-text intent into a structured action via an LLM extraction prompt, then executes the action against the wiki and returns a human-readable result. When the intent is ambiguous (e.g. \"activate a page\" without specifying which), sets `needs_clarification=True` and returns a prompt and candidate list for the web UI to render as chip buttons.\n\n### ScaffoldAgent\n\nGenerates and updates the wiki's structural pages. Reads all current wiki pages and asks the LLM to organise them into 5–8 domain-appropriate categories, then writes `wiki/index.md` (category index with wikilinks), `wiki/purpose.md` (scope statement), and `AGENTS.md` (ingest and query guidelines). Also stamps a `categories:` field on each page's YAML frontmatter. When `ROUTING.md` already exists, regenerates it to stay in sync with the updated index structure.\n\n### RewriteAgent\n\nRewrites follow-up questions in multi-turn conversations into self-contained, standalone form before BM25 retrieval. Converts context-dependent phrases (\"What came after that?\", \"Tell me more about his early life\") into explicit questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the correct pages without relying on conversation context. Only invoked when conversation history is non-empty. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### SummarizeAgent\n\nCompresses the oldest conversation turns into a single `[Session summary: …]` assistant turn when a session exceeds `conversation_history_turns`. Prevents unbounded context growth across long sessions. Emits a `notice` SSE event the first time compression occurs so the user can see that earlier context was condensed. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### ContextAgent\n\nBuilds token-budgeted context packs for MCP tool consumers. See [Section 20 — Context Packs](#20-context-packs) for full detail.\n\n### SearchDecomposeAgent\n\nDecomposes a web search intent into 1–4 terse keyword search strings optimised for authoritative source retrieval. Uses a separate prompt from `QueryAgent`'s question decomposition — search decomposition asks \"what distinct search strings would find the best sources?\" (terse keyword phrases) rather than \"what distinct questions does this ask?\" (natural-language sub-questions). See [Web search fan-out](#web-search-fan-out) in the IngestAgent section.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n| `session` | `.jsonl` | `claude session`, `codex session`, `cursor session`, `ai session`, `session history` | Extracts human-readable turns from AI coding session transcripts. Supports Claude Code JSONL format and Codex/Cursor format. Filters tool calls, thinking blocks, and sub-agent scaffolding. Short turns (< 20 assistant words, < 3 user words) are skipped. No external dependencies. |\n\n### Session Skill — AI Session History Ingestion\n\nThe `session` skill turns AI coding session history files (`.jsonl`) into searchable wiki pages capturing the problem-solving knowledge exchanged during the session.\n\n**Supported formats**\n\n| Format | File origin | Detection |\n|--------|-------------|-----------|\n| Claude Code | `~/.claude/projects//.jsonl` | `obj.type` ∈ `{\"user\", \"assistant\"}` with nested `obj.message` |\n| Codex / Cursor | Export from OpenAI Codex or Cursor IDE | `obj.role` at top level, no `message` wrapper |\n\nFormat is auto-detected from the first 30 parseable lines. Files that match neither format are parsed as Codex (fallback).\n\n**Filtering rules**\n\n| Content | Kept? | Reason |\n|---------|-------|--------|\n| User text ≥ 3 words | ✓ | Substantive question or instruction |\n| Assistant text ≥ 20 words | ✓ | Substantive answer |\n| User text < 3 words (`\"ok\"`, `\"yes\"`) | ✗ | Too terse to be useful |\n| Assistant text < 20 words | ✗ | Acknowledgement or one-liner |\n| `isSidechain: true` lines | ✗ | Sub-agent scaffolding (not the user's conversation) |\n| `tool_use` blocks | ✗ | Shell commands / file writes — not final output |\n| `tool_result` blocks | ✗ | Raw output — avoids leaking file contents or credentials |\n| `thinking` blocks | ✗ | Internal reasoning — not the final answer |\n| `permission-mode`, `system`, `last-prompt` lines | ✗ | Session metadata, not conversation |\n\nAfter extraction the text passes through Synthadoc's standard pre-LLM source sanitizer\n(zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars,\ninstruction-override phrases) — the same step applied to every PDF, DOCX, URL, and other source type.\nSee [§29 Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer).\n\n**Output format**\n\nEach kept turn is labelled and separated by `---`:\n\n```\n[USER]\nHow do I implement a sliding window algorithm in Python?\n\n---\n\n[ASSISTANT]\nA sliding window algorithm maintains a contiguous subarray by advancing two\npointers simultaneously…\n```\n\n**`suggested_slug`**\n\nThe skill sets `metadata[\"suggested_slug\"]` to `session-YYYY-MM-DD-` where the date comes from the file's `mtime` and the topic from the first substantive user turn (first 6 words, slugified). Example: `session-2026-07-15-how-do-i-implement-sliding`.\n\n**Limitations**\n\n- No chunking in v1.1 — very long sessions are truncated at `max_source_chars` (default 400 000 chars). Split large archives into individual `.jsonl` files.\n- Tool output is excluded by design — re-ingest the original source files if you need the file contents in the wiki.\n- Re-ingesting the same `.jsonl` file uses Synthadoc's standard source-hash dedup — no duplicate pages are created.\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n**Compound identifier expansion:** When a token contains underscores, the tokenizer emits both the compound form and each component as separate tokens (`capex_growth` → `[\"capex_growth\", \"capex\", \"growth\"]`). This expansion is applied at both index time and query time so that human-friendly queries match programmatic identifiers and vice versa. Leading/trailing underscores are stripped before splitting; no empty strings are emitted.\n\n**TF fallback for small corpora:** When all BM25 scores in a result set are zero or negative (IDF collapse on a corpus of ≤ 2 documents, or a term present in every page), the search system falls back to normalised term frequency (TF) scoring: `TF(doc, query) = Σcount(term, doc) / len(doc_tokens)`. Results are returned in the same format as BM25 results with a `tf_fallback: True` flag on each `SearchResult`. This field is recorded in the audit trail and consumed by gap detection (see below).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp, cascade_links_removed_from: [str]}` _(cascade field added v1.0.2)_ |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n| `GET` | `/graph` _(v1.0.0)_ | — | `{status, node_count, edge_count, cluster_count, nodes: [...], edges: [...]}` or `{status: \"computing\"}` on first call |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n**External file paths** (outside the wiki root) are supported when the request comes from\n`127.0.0.1` or `::1` and the payload includes `\"allow_external_paths\": true`. The CLI sets\nthis flag automatically for all local file sources. Remote clients cannot set this flag\n(the server ignores it for non-localhost connections) to prevent arbitrary file reads\nin server-exposed deployments.\n\nThis is required for ingesting files that live outside the wiki directory by design —\nfor example, Claude Code session transcripts at `~/.claude/projects//.jsonl`.\n\n**Client behaviour summary:**\n\n| Client | Sends `allow_external_paths` | Can ingest outside wiki root? |\n|--------|------------------------------|-------------------------------|\n| CLI (`synthadoc ingest `) | `true` (automatic, local file paths only) | Yes — server is localhost |\n| Obsidian plugin | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Web UI | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Direct HTTP API (localhost) | Set manually in request body | Yes — if request from 127.0.0.1/::1 |\n| Direct HTTP API (remote) | Ignored by server | No — server silently treats as `false` |\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 1.1.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n**Reading View default:** `synthadoc plugin install` and `synthadoc plugin upgrade` write `\"defaultViewMode\": \"preview\"` to `.obsidian/app.json` after copying the plugin files. This causes Obsidian to open new notes in Reading View by default, which is required for `^[filename:L-L]` citation chips to render. Only `defaultViewMode` is written; all other keys in `app.json` are preserved. If `app.json` is malformed JSON the write is skipped and a warning is logged — the install still succeeds.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports two things: dangling slugs (pages listed in ROUTING.md that no longer exist in the wiki) and unassigned slugs (pages that exist in index.md but are not listed in any ROUTING.md branch). Enabled only when ROUTING.md exists. **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n| `Graph: show knowledge graph` _(v1.1.0)_ | Open the knowledge graph panel — draggable modal with Canvas force-directed graph, type filter dropdown, cluster legend, hover tooltip showing title/slug/type/state/cluster/connections, and click-to-open page in current pane. System pages (index, overview, dashboard, purpose, log) are excluded. Capped at 300 most-connected nodes for large wikis with an explanatory banner. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [-w wiki] — generate ROUTING.md from index.md branch structure\n│ ├── validate [-w wiki] — report dangling slugs and cross-branch duplicates\n│ └── clean [-w wiki] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [-w wiki]\n├── candidates\n│ ├── list [-w wiki]\n│ ├── promote |--all [-w wiki]\n│ └── discard |--all [-w wiki]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** subscription, or want to use the free **Opencode Zen** tier, you can use either as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode Zen (free, no API key — requires connecting first: run `opencode` → `/connect` → select Zen):\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"opencode/big-pickle\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n# Citation pass (Pass 4) tuning — these two settings work together:\n# citation_source_lines — how many lines of the source the LLM sees when placing ^[...] markers.\n# Increase if lint reports out_of_range on long sources (transcripts, PDFs).\n# citation_max_tokens — output token budget for the annotated section returned by the LLM.\n# Increase if you raise citation_source_lines and have long wiki sections.\n# citation_source_lines = 400\n# citation_max_tokens = 8192\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax`, `qwen` (DashScope), and `deepseek` providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Emit `logger.warning` in server log when per-job ingest cost exceeds this threshold; job continues normally |\n| `cost.hard_gate_usd` | float | `2.00` | Permanently fail the ingest job (DEAD) and record a `cost_gate_exceeded` audit event when per-job cost exceeds this threshold |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × (backup_count + 1)` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n| `ingest.max_source_chars` | int | `32000` | Character limit applied to each source before the LLM call. Sources exceeding this limit are truncated; the page's `sources:` frontmatter entry gets `truncated: true` and lint emits a warning. Override per-run with `--max-source-chars N`. _(v1.0.0)_ |\n| `ingest.citation_source_lines` | int | `400` | Number of source lines the LLM sees during Pass 4 (citation annotation). Increase if lint reports `out_of_range` on long sources such as transcripts or PDFs. Raise `citation_max_tokens` in proportion when increasing this value. |\n| `ingest.citation_max_tokens` | int | `8192` | Output token budget for the annotated section returned by Pass 4. Increase when `citation_source_lines` is raised or wiki sections are long, to avoid truncated annotations. |\n| `query.context_wiki_pct` | float | `0.60` | Fraction of the model context window reserved for wiki source pages. _(v1.0.0)_ |\n| `query.context_history_pct` | float | `0.20` | Fraction reserved for conversation history. _(v1.0.0)_ |\n| `query.context_system_pct` | float | `0.15` | Fraction reserved for system prompt and instructions. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_index_pct` | float | `0.05` | Fraction reserved for the wiki index preamble. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_window` | int | _(auto)_ | Override the built-in context window lookup for the configured model. Use when running a local or custom model whose window size is not in the built-in table. _(v1.0.0)_ |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement is wired into the ingest pipeline: `Orchestrator._guard_ingest_cost()` is called after each ingest LLM call with the returned cost, before the job is marked complete.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | `logger.warning` in server log; job continues normally |\n| `hard_gate_usd` | $2.00 | Server/ingest path: records `cost_gate_exceeded` audit event + permanently fails job (DEAD). Interactive CLI: prompts `Proceed? [y/N]` |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` by default. The bind address is configurable via `server.host` in `config.toml` (e.g. `host = \"0.0.0.0\"` to expose on all interfaces for LAN access). No built-in authentication — restrict via firewall when exposing beyond loopback. Remote access without a reverse proxy is not recommended on shared or networked machines.\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs and unassigned slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `-w ` / `--wiki ` to target a specific wiki.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; when all local source files are missing from disk the page is transitioned to `archived` (if only some sources are missing it is transitioned to `stale` instead). Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: lint evaluates all local (non-URL) source paths listed in the page's frontmatter. If **all** of them are missing from disk, the page is transitioned to `archived`. If only **some** are missing — meaning valid content from at least one source still exists — the page is transitioned to `stale` instead, preserving it for re-ingest rather than discarding it. _(v1.0.2: previously any single missing source triggered an immediate archive.)_\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Cascade link cleanup\n\nWhenever a page transitions to `archived` — whether triggered manually (`synthadoc lifecycle archive`), via the Obsidian Lifecycle modal, the MCP `synthadoc_lifecycle` tool, or automatically by lint detecting a missing source — Synthadoc immediately scans all other active pages and removes every `[[archived-slug]]` wikilink pointing to the now-archived page.\n\n- **Inline links** — `[[archived-slug]]` becomes plain text (the display text if an alias was used, otherwise the slug itself).\n- **List-item links** — a list item whose only content is `[[archived-slug]]` is dropped entirely.\n- **Archived pages are skipped** — pages already archived are not rewritten.\n- **System pages are skipped** — index, overview, dashboard, log, and purpose pages are excluded.\n\nThe cleanup runs synchronously before the response is returned, so the wiki is consistent immediately. The CLI and MCP responses include a `cascade_links_removed_from` field listing the affected slugs. During `lint run`, cascade is batched once after all auto-archive transitions complete, and the count is added to `dangling_links_removed` in the lint report.\n\nLint's existing periodic dangling-link cleanup (`lint run --scope orphans`) remains in place as a safety net for any dead links that predate this feature or arrive through other paths.\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [...], \"input_tokens\": 1240, \"output_tokens\": 380, \"tokens_used\": 1620}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […], \"input_tokens\": N, \"output_tokens\": N, \"tokens_used\": N}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Streaming Token Audit\n\nStreaming LLM responses do not return token counts in the same way as blocking calls. To track real usage in the audit trail:\n\n**Provider-side collection** — each `LLMProvider` subclass exposes two instance attributes: `last_stream_input_tokens` and `last_stream_output_tokens`. These default to `0` and are populated from the API-specific usage field when the stream ends:\n\n| Provider | Mechanism | Verified |\n|---|---|---|\n| OpenAI | `stream_options={\"include_usage\": True}` passed to `chat.completions.create()`; final chunk carries `chunk.usage.prompt_tokens` / `completion_tokens` with empty `choices` | ✅ |\n| Anthropic | `message_start` event → `event.message.usage.input_tokens`; `message_delta` event → `event.usage.output_tokens` | ✅ |\n| Ollama | Final chunk with `done=True` → `prompt_eval_count` / `eval_count` | ✅ |\n| DeepSeek | Same `OpenAIProvider` path as OpenAI; DeepSeek's OpenAI-compatible API supports `stream_options` | ✅ |\n| MiniMax (reasoning: M2.5+) | Detects `` in stream → falls back to blocking `complete()` → captures exact counts from `resp.usage` | ✅ |\n| MiniMax (non-reasoning, e.g. M3 thinking=disabled) | Same `OpenAIProvider` path; MiniMax API silently ignores `stream_options`. Falls back to character-based estimate (÷ 3.5 chars/token) from prompt + answer lengths. Accuracy ±20%. | ✅ estimated |\n| Gemini | Same `OpenAIProvider` path via `generativelanguage.googleapis.com/v1beta/openai/`; Google's compatibility layer honours `stream_options` — verified live (Gemini 2.5 Flash Lite, 50K tokens) | ✅ |\n| Groq | Same `OpenAIProvider` path; Groq's OpenAI-compatible API supports `stream_options` | ✅ |\n| Qwen (DashScope) | Same `OpenAIProvider` path via DashScope; `stream_options` honoured — verified live (qwen-plus, 28K tokens) | ✅ |\n| Qwen (Ollama) | Ollama path → `prompt_eval_count` / `eval_count` | ✅ |\n\n**Forwarding to `done` event** — after `complete_stream()` is exhausted, `QueryAgent.run_stream()` reads the provider attributes and includes them in the final `done` event payload: `input_tokens`, `output_tokens`, `tokens_used`.\n\n**Audit record** — `Orchestrator.query_stream()` consumes the `done` event and calls `estimate_cost(model, input_tokens, output_tokens, is_local)` from `synthadoc/providers/pricing.py`, then calls `AuditDB.record_query()` with the real token count and computed cost. The audit row at `GET /audit/queries` will therefore always show non-zero values for cloud providers.\n\nWhen a provider does not report streaming usage (e.g. a future provider not yet wired), the attributes stay at `0`; the audit row records `tokens=0, cost_usd=0.0` rather than crashing, matching the pre-fix behaviour.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 5) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 5 # turns to include in each request (default: 5; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp, cascade_links_removed_from: [str]}` or `{error, cascade_links_removed_from: []}` _(cascade field added v1.0.2)_ | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions for Codex CLI / OpenCode (if present)\n├── CLAUDE.md ← same content; loaded by Claude Code (if present)\n├── GEMINI.md ← same content; loaded by Gemini CLI (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── hooks/ ← user hook scripts (always included)\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ ├── extracted/ ← text sidecars + PDF pagemaps (always included)\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `logs/`.\n\n> **v0.9.3 fix:** `hooks/` and `.synthadoc/extracted/` were previously missing from backups. Omitting `extracted/` caused the Source Viewer and Provenance citation modal to show \"Could not read source file\" after restore.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## 29. Pre-LLM Source Sanitizer\n\nEvery source document passes through a sanitizer immediately after text extraction, before any LLM call. The sanitizer removes six categories of content that could manipulate the LLM or degrade compilation quality:\n\n| Category | Action | Warning logged? |\n|---|---|---|\n| Zero-width characters (U+200B, U+200C, U+200D, U+FEFF) | Removed silently | No |\n| Bidi override characters (U+202A–U+202E, U+2066–U+2069) | Removed | Yes |\n| HTML comments (``) | Removed silently | No |\n| Hidden CSS spans (`display:none`, `visibility:hidden`) | Removed silently | No |\n| Base64 blobs ≥ 200 consecutive characters | Replaced with `[base64 content removed]` | Yes |\n| Instruction-override phrases (8 patterns: \"ignore previous instructions\", \"disregard the above\", \"override your system prompt\", etc.) | Replaced with `[redacted]` | Always |\n\nWhen content is removed and a warning is appropriate, the server logs a `WARN` entry:\n```\n[WARN] sanitizer stripped content from 'papers/survey.pdf': bidi overrides, instruction-override phrase\n```\n\nThis is a pure pre-processing step with zero LLM cost. Truncation (if configured) is applied after sanitization, so the character budget is consumed by clean text only.\n\n---\n\n## 30. Per-Source Truncation Flag\n\nLarge source documents are truncated before they reach the LLM. By default the limit is **32,000 characters** (~8,000 tokens), raised from the prior hardcoded 8,000-character limit. When a source exceeds this limit, its compiled wiki page records a `truncated: true` flag in its `sources:` frontmatter entry.\n\n### Configuration\n\n```toml\n[ingest]\nmax_source_chars = 32000 # default; raise for large PDFs or books\n```\n\n### Per-ingest override\n\nThe limit can be raised for a single ingest run without modifying the config:\n\n```bash\nsynthadoc ingest papers/large-textbook.pdf --max-source-chars 128000\n```\n\nThe same `max_source_chars` field is accepted by the HTTP `POST /jobs/ingest` body and the MCP `synthadoc_ingest` tool.\n\n### Lint surfacing\n\n`synthadoc lint` emits a warning for any page that has a truncated source:\n\n```\n[WARN] quantum-computing.md: source 'papers/large-survey.pdf' was truncated at ingest\n (source exceeded max_source_chars=32000 — 87,412 chars in source).\n To re-ingest with a higher limit (this source only):\n synthadoc ingest papers/large-survey.pdf --max-source-chars 128000\n To raise the limit for all future ingests:\n set [ingest] max_source_chars = 128000 in your config\n```\n\n---\n\n## 31. Proportional Context Budget\n\nQuery context is allocated proportionally to the configured model's context window, replacing the prior hard cap. The default allocation:\n\n| Slice | Default | Purpose |\n|---|---|---|\n| `context_wiki_pct` | 60% | Wiki page content (ranked by BM25 + vector score) |\n| `context_history_pct` | 20% | Chat turn history (newest-first, oldest dropped when over budget) |\n| `context_system_pct` | 15% | System prompt and purpose document |\n| `context_index_pct` | 5% | `ROUTING.md` and search index |\n\nPercentages must sum to ≤ 100. Configure in `[query]`:\n\n```toml\n[query]\ncontext_window = 0 # 0 = auto-detect from known model map\ncontext_wiki_pct = 60\ncontext_history_pct = 20\ncontext_system_pct = 15\ncontext_index_pct = 5\n```\n\n> **Note:** `context_system_pct` and `context_index_pct` are parsed and validated but not yet enforced as hard caps in this release — the system prompt uses a conservative fixed limit, which is well within the configured 15% slice for all supported models. Full per-slice enforcement is planned for v1.1.\n\n### Model context window map\n\nSynthadoc ships a built-in prefix-matched table:\n\n| Model | Context window |\n|---|---|\n| `claude-opus-4*`, `claude-sonnet-4*`, `claude-haiku-4*` | 200,000 tokens |\n| `gpt-4o*`, `gpt-4-turbo*` | 128,000 tokens |\n| `gpt-4` (exact) | 8,192 tokens |\n| `gpt-3.5-turbo*` | 16,385 tokens |\n| Unknown / fallback | 128,000 tokens |\n\nSet `context_window = N` in config to override the map (useful for local Ollama models with non-standard context sizes).\n\nAt the defaults on a 200k-token model: wiki slice ≈ 480,000 chars — far more than any practical query needs. On constrained models, the budget scales down automatically so the same config works across model families.\n\n---\n\n## 32. Knowledge Graph\n\nSynthadoc computes a wikilink graph across all active and draft pages during every lint run. The graph is stored in two tables in `audit.db`:\n\n```sql\nCREATE TABLE graph_nodes (\n slug TEXT PRIMARY KEY,\n cluster_id INTEGER NOT NULL,\n updated_at TEXT NOT NULL\n);\n\nCREATE TABLE graph_edges (\n from_slug TEXT NOT NULL,\n to_slug TEXT NOT NULL,\n weight INTEGER NOT NULL DEFAULT 1,\n edge_type TEXT NOT NULL DEFAULT 'mixed',\n PRIMARY KEY (from_slug, to_slug)\n);\n```\n\nEdge weight is a composite signal from two sources:\n\n| Edge type | Source | Weight contribution |\n|-----------|--------|---------------------|\n| `wikilink` | `[[slug]]` occurrences in page body | +1 per occurrence |\n| `co_source` | pages compiled from the same source file (matched by hash) | +2 per shared source |\n\nMost edges are `mixed` (have both types). Pure `co_source` edges surface hidden relationships — pages compiled from the same source document are linked immediately after ingest, before any wikilinks are created. `edge_type` is stored on each edge so the web UI can render co-source edges with a different visual style.\n\nCluster IDs are assigned by Louvain community detection — pages that link densely to each other end up in the same cluster.\n\n### REST API\n\n**`GET /graph`** — returns the full graph with per-node enrichment:\n\n```json\n{\n \"status\": \"ready\",\n \"node_count\": 45,\n \"edge_count\": 123,\n \"cluster_count\": 4,\n \"nodes\": [\n {\"slug\": \"quantum-error-correction\", \"title\": \"Quantum Error Correction\",\n \"type\": \"concept\", \"state\": \"active\", \"cluster_id\": 2}\n ],\n \"edges\": [\n {\"from\": \"quantum-error-correction\", \"to\": \"shor-algorithm\", \"weight\": 3, \"edge_type\": \"mixed\"}\n ]\n}\n```\n\nWhen the graph tables are empty (first call after upgrade, or before the first lint run), the server returns `{\"status\": \"computing\"}` and starts a background build immediately. The web UI polls every 2 seconds and renders the graph when the status becomes `\"ready\"`.\n\n### Web UI\n\nThe **Graph** tab (alongside Chat in the top nav) renders a D3.js force-directed graph. Nodes are colored by cluster. Edge thickness is proportional to weight (1–4 px). Pure `co_source` edges (no wikilink) are rendered with a dashed stroke, making hidden source-based relationships visually distinct from explicit cross-links. Clicking a node opens a sidebar showing the page title, type and lifecycle state badges, cluster assignment, and an **\"Ask about this →\"** button that switches to the Chat tab with a pre-filled query.\n\nControls: zoom in/out (scroll or pinch), drag nodes, filter by page type.\n\n### System page exclusion _(v1.1.0)_\n\nThe five structural system pages — `overview`, `index`, `dashboard`, `purpose`, and `log` — are excluded from both `graph_nodes` and `graph_edges`. These pages link to nearly every other page and would act as false hubs, inflating edge counts and distorting Louvain cluster detection. System page exclusion is applied inside `_build_graph()` before any edge computation.\n\n### Obsidian knowledge graph panel _(v1.1.0)_\n\n→ Full description: [§33 Multi-Platform Agent Skill Files](#33-multi-platform-agent-skill-files) covers scaffold; the graph panel itself is documented in [§8 Obsidian Plugin](#8-obsidian-plugin) command table.\n\nThe Obsidian plugin includes a **draggable modal** knowledge graph panel (`Graph: show knowledge graph` command). It fetches from the same `GET /graph` endpoint as the web UI and renders a force-directed Canvas 2D simulation using Verlet integration (no external library). System pages are already excluded from the server response.\n\n| Feature | Obsidian panel | Web UI Graph tab |\n|---------|----------------|-----------------|\n| Engine | Canvas 2D + Verlet | D3.js SVG |\n| Node click | Opens the wiki page in the current pane | Opens the node sidebar with \"Ask about this →\" |\n| **Best for** | **Navigating to and editing pages** | **Asking questions about a page** |\n| Draggable panel | Yes — drag header to reposition | Fixed tab layout |\n| Node cap | 300 most-connected (see note) | No cap |\n| Cluster legend | Yes | Yes |\n| Type filter | Yes | Yes |\n\n**Rule of thumb:** Use the **web UI** when you want to ask questions about a node. Use the **Obsidian panel** when you want to navigate to and edit a page.\n\n**Node cap rationale:** The Obsidian panel uses a hand-written Verlet integrator with O(n²) pairwise charge repulsion — every node repels every other node on each animation tick. At 300 nodes that is ~90,000 operations per frame, which runs smoothly at 60 fps inside Obsidian's Electron renderer. At 500 nodes it is ~250,000 operations per frame and starts to lag noticeably. The web UI has no cap because D3's `forceManyBody` uses a Barnes-Hut tree approximation (O(n log n)), which scales to thousands of nodes without frame-rate issues. When the cap is hit, nodes are ranked by degree and the 300 most-connected are shown; a banner reports the total page count and the number of hidden nodes, and suggests using the type filter to bring the set below 300.\n\n→ User walkthrough: [Quick-Start Guide §24 — Knowledge Graph](docs/user-quick-start-guide.md#knowledge-graph)\n\n---\n\n## 33. Multi-Platform Agent Skill Files\n\n\n\n`synthadoc init` and `synthadoc scaffold` write three agent skill files at the wiki root:\n\n| File | Format | Read by |\n|------|--------|---------|\n| `AGENTS.md` | OpenAI Agents SDK / Codex / OpenCode | Codex CLI, OpenCode, generic agents |\n| `CLAUDE.md` | Claude Code / Anthropic | Claude Code (loaded automatically when the wiki folder is opened) |\n| `GEMINI.md` | Gemini CLI | Gemini CLI |\n\nAll three files carry **identical content**: a title block, the LLM-generated domain guidelines for the wiki, a quick-reference CLI command table, and an MCP tool table. The only difference is the heading comment. This means any AI coding tool that opens the wiki root gets full Synthadoc guidance without manual setup.\n\n### Content generated by `scaffold`\n\nEach skill file is generated from the same LLM call that produces `purpose.md` and `index.md`. The LLM receives the current wiki page list and domain description, then generates:\n\n1. **Domain label** — a precise human-readable name inferred from wiki content (e.g. `History of Computing`, not `My Wiki`)\n2. **Domain guidelines** — 3 domain-specific ingest and query guidelines as plain sentences\n3. **Quick Reference table** — pre-filled with the current wiki name and port\n\nIf the wiki was created before v1.1.0, running `synthadoc scaffold` once regenerates all three files and adds `CLAUDE.md` and `GEMINI.md` to any wiki that only had `AGENTS.md`.\n\n### Template source\n\nThe instruction body is defined in `synthadoc/cli/_init.py` as `_AGENT_INSTRUCTION_BODY` and shared by all three templates, keeping the files permanently in sync. The scaffold agent substitutes `{domain}`, `{guidelines}`, and `{port}` at write time.\n\n---\n\n## 34. Synthadoc Scaffold Marker\n\n\n\nThe scaffold marker `` (written as an HTML comment) separates **user-authored content** (preserved) from **scaffold-managed content** (rewritten on each `synthadoc scaffold` run).\n\n### Single-marker mode (index.md)\n\nThe marker appears once, below the H1 heading in `index.md`. Everything above the marker is the user zone; everything below is replaced with new LLM-generated categories.\n\n```markdown\nMy custom introduction — maintained by hand.\n\n\n\n## Pioneers and Visionaries\n- [[alan-turing]]\n...\n```\n\nIf the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### Per-section multi-marker mode (purpose.md)\n\n`purpose.md` uses a **per-section** approach: the marker appears inside each `## Section` heading. Within a section, content the user writes **above** the marker is preserved; content below is replaced with fresh LLM output. Sections without a marker in the existing file are replaced entirely (and a marker is added). Sections in the existing file that the scaffold template does not include are kept unchanged.\n\n```markdown\n## Overview\n\nMy custom overview note.\n\n\n\nLLM-generated overview content goes here.\n\n## What Belongs\n\n\n\nLLM-generated scope definition.\n```\n\nThis design lets users annotate any section of `purpose.md` without losing their notes on the next scaffold run.\n\n### Marker preservation guarantee\n\n- `preserve_user_zone()` in `scaffold_agent.py` handles both modes.\n- If a section has no marker in the existing file, `preserve_user_zone()` treats the entire section body as user content and preserves it, replacing only what falls below any marker it finds.\n- The scaffold NEVER touches `config.toml`, `dashboard.md`, or any wiki page (`wiki/*.md`).\n\n→ User walkthrough: [Quick-Start Guide §14 — Enrich the wiki with scaffold](docs/user-quick-start-guide.md#scaffold)\n\n---\n\n## Customization\n\nSynthadoc exposes four extension points — all hot-loaded, no server restart required.\n\n### Custom skills (new file formats)\n\nSubclass `BaseSkill` (Apache-2.0 — no AGPL obligation on your skill code), drop the folder in `/skills/` or `~/.synthadoc/skills/`, and Synthadoc picks it up on the next ingest. Skills match by file extension or intent prefix and support any Unicode text, including CJK prefixes.\n\n→ Full interface, manifest format, and examples: [§5 Skills System](#5-skills-system) and [§17 Plugin Development Guide](#17-plugin-development-guide)\n\n### Custom LLM providers\n\nSubclass `LLMProvider` from `synthadoc/providers/base.py` (Apache-2.0) and place the file in `~/.synthadoc/providers/` or the wiki `providers/` directory. Switch active provider with one config line.\n\n→ Full interface and wiring instructions: [§10 Configuration — Provider switching](#10-configuration)\n\n### Hooks\n\nShell scripts (any language) that fire on `on_ingest_complete` and `on_lint_complete`. Receive a JSON context on stdin. Set `blocking = true` to gate the operation on the hook's exit code — useful for CI pipelines and quality gates.\n\n→ Full event schema, context JSON, and blocking behaviour: [§11 Hook System](#11-hook-system)\n\n### Cache control\n\nThree cache layers (embedding, LLM response, provider prompt cache) invalidate automatically on source-file change (SHA-256). Force a fresh call with `--force`, or wipe all cached responses with `synthadoc cache clear -w `.\n\n→ Layer-by-layer breakdown and invalidation rules: [§12 Cache System](#12-cache-system)\n\n### Multi-Platform Agent Skill Files\n\nEvery Synthadoc wiki ships three companion files that give AI coding agents a complete, self-contained operating guide for the wiki:\n\n| File | Platform |\n|---|---|\n| `AGENTS.md` | Codex CLI, OpenCode, and any agent that reads the AGENTS.md convention |\n| `CLAUDE.md` | Claude Code (this session's tool) — highest-priority instruction source in Claude's hierarchy |\n| `GEMINI.md` | Gemini CLI — relevant because Synthadoc's default LLM provider is `gemini-2.5-flash-lite` |\n\nAll three files share identical body content generated from the same template; they differ only in their H1 heading so each agent recognises its own file first. The body covers:\n\n- **Domain guidelines** — LLM-generated bullet list of domain-specific ingest and query rules (produced by `scaffold`)\n- **Quick reference table** — every common CLI command with the correct `synthadoc` syntax\n- **Server startup** — how to start the server and verify it is running\n- **Ingest examples** — local file, URL, YouTube, agent session `.jsonl`, force reingest, dry-run\n- **Query** — basic and streaming query syntax; citation marker format\n- **Lint** — when to run and what each check does\n- **Lifecycle** — transition commands with state names\n- **Page schema** — frontmatter structure with all fields\n- **MCP tools table** — all 12 tools with purpose descriptions\n\n**How the files are created and updated:**\n\n1. `synthadoc init` — writes all three files with default domain guidelines and the wiki's configured port.\n2. `synthadoc scaffold` — regenerates all three with LLM-produced domain guidelines derived from the current wiki state. The `scaffold` command prints a confirmation line for each file.\n\n**Use cases:**\n\n- Open Claude Code in a Synthadoc wiki directory and it immediately knows how to ingest, query, lint, and manage lifecycle — no additional setup. Claude Code can also start the server in background mode (`synthadoc serve -w -b`), stop it cleanly, and orchestrate multi-step workflows, effectively acting as a control plane for the wiki.\n- Gemini CLI reads `GEMINI.md` and can run `synthadoc` commands with the correct flags without asking the user for help.\n- CI pipelines with Codex CLI use `AGENTS.md` to drive automated lint and lifecycle transitions on PR merge.\n- Agent session files produced by Claude Code (`.jsonl` under `~/.claude/projects/`) can themselves be ingested into the wiki, capturing the reasoning behind design decisions alongside the source documents.\n\n→ See [README.md — Interfaces & Integration](../README.md#interfaces--integration) for the feature comparison row and quick-start instructions.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v1.1.0\n\n- **Weighted knowledge graph edges** — graph edges now carry two signals: wikilink occurrences (+1 each) and co-source connections — pages compiled from the same source file (matched by SHA-256 hash) — (+2 per shared source). `edge_type` field added to `graph_edges` table (`wikilink`, `co_source`, or `mixed`). `GET /graph` exposes `edge_type` per edge. Web UI renders edge thickness proportional to weight (1–4 px) and dashed lines for pure co-source edges. Co-source edges appear immediately after ingest — before any wikilinks — surfacing hidden relationships in the graph. Schema migrated to version 3.\n- **Multi-platform agent skill files** — `synthadoc init` and `synthadoc scaffold` write `CLAUDE.md`, `AGENTS.md`, and `GEMINI.md` at wiki root; all three carry LLM-generated domain guidelines and the full CLI quick-reference so Claude Code, Codex CLI, OpenCode, Pi, and Gemini CLI users get first-class guidance without manual setup. Run `synthadoc scaffold` once on any pre-v1.1.0 wiki to generate the new files. See [§33](#33-multi-platform-agent-skill-files).\n- **Session history ingestion skill** — `.jsonl` session files from Claude Code and Codex CLI are ingested as wiki pages; human turns and substantive assistant responses extracted; tool calls, thinking blocks, and sub-agent scaffolding skipped; format auto-detected; `suggested_slug` derived from session date and first user message. Converts ephemeral problem-solving knowledge into searchable wiki entries. See [§5 Skills System — Session Skill](#session-skill--ai-session-history-ingestion).\n- **Obsidian knowledge graph panel** — new `Graph: show knowledge graph` command in the Obsidian plugin opens a draggable modal with a Canvas 2D force-directed graph. Features: Verlet integration, type filter dropdown, cluster legend, hover tooltip (title, slug, type, state, cluster, connections), click-to-open page in current pane, node drag to repin. Closes only via the × button — Escape and outside-click do not close the panel, matching the behaviour of other Synthadoc modals. **Large-wiki cap:** when the node set exceeds 300 (after any type filter is applied), nodes are ranked by degree (total edge count) and the 300 most-connected are displayed; edges to capped-out nodes are removed; a banner explains the cap and suggests using the type filter to narrow the view below 300. See [§8 Obsidian Plugin](#8-obsidian-plugin) and [§32 Knowledge Graph — Obsidian panel](#obsidian-knowledge-graph-panel-v110).\n- **Per-section scaffold markers in `purpose.md`** — `purpose.md` now uses per-section `` markers. Users can write content above the marker in any `## Section` block; that content is preserved on every re-scaffold. A single-marker mode for `index.md` continues to work as before. See [§34 Synthadoc Scaffold Marker](#34-synthadoc-scaffold-marker).\n- **System pages excluded from knowledge graph** — `overview`, `index`, `dashboard`, `purpose`, and `log` are filtered out of `_build_graph()` before any edge computation so they cannot act as false hubs in Louvain cluster detection. Both the Obsidian graph panel and the web UI Graph tab reflect this exclusion.\n- **Web UI graph cluster legend** — the web UI Graph tab now shows a cluster legend below the canvas (matching the Obsidian plugin panel), with a color swatch and label for each Louvain cluster present in the filtered node set.\n- **`_merge_section_into_page` deduplication** — force re-ingest of the same source file (e.g. an updated `.jsonl` session) now replaces the page body rather than appending a duplicate section. `_files_match()` normalizes absolute vs. relative paths so the same file is recognized regardless of how it was passed to the ingest command.\n- **`purpose.md` domain label inference** — `scaffold` now infers a precise `domain_label` from current wiki content (e.g. `History of Computing`) instead of using a generic placeholder; the label is used in all three agent skill files and in `purpose.md`'s Overview section.\n- **CI: Windows runner fix** — `scripts/sync_plugin.py` and `scripts/sync_web_ui.py` now use `shutil.copy2` instead of `cp -f`, resolving CI failures on Windows runners for the sync check.\n\n### v1.0.2\n\n- **Cascade link cleanup on archive** — archiving any page immediately removes every `[[slug]]` wikilink pointing to it from all other active pages, without waiting for the next `lint run`. Inline links become plain text; list-item-only links are dropped entirely. The HTTP `POST /lifecycle/transition` and MCP `synthadoc_lifecycle` responses now include a `cascade_links_removed_from` field listing the rewritten page slugs. During `lint run`, cascade runs once after all auto-archive transitions complete and the count is included in the lint report's `dangling_links_removed` field. System pages (index, dashboard, purpose, log, overview) and already-archived pages are skipped. The existing dangling-link lint pass remains as a safety net for dead links that predate this release.\n- **Multi-source partial-archive fix** — when a page has multiple local source files and only some are missing from disk, lint now transitions the page to `stale` (not `archived`), preserving it for re-ingest. The page is only archived when every local source file is gone. URL sources are not affected by this check.\n\n### v1.0.1\n\n- **Scaffold JSON extraction hardened** — replaced the greedy regex used to extract the LLM's JSON response with a brace-balanced parser; eliminates false-positive matches on `[[wikilink]]` syntax that previously caused `json.loads()` failures and silent scaffold corruption\n- **Generated file validation** — scaffold now validates every LLM-generated file (index, purpose, AGENTS.md) for minimum length, required frontmatter, and `[[wikilink]]` presence before writing to disk; validation failure marks the job as failed with a structured error message rather than silently writing a malformed page\n- **ROUTING.md regeneration validation** — `routing init` and scheduled routing regeneration now validate the regenerated file for minimum line count and anchor presence before applying it; prevents a corrupt or truncated ROUTING.md from entering the wiki\n- **Scaffold failure reporting** — scaffold skip and failure reasons are now surfaced in the terminal output and job error field; previously silent failures (e.g. LLM timeout, JSON parse error) are now visible without tailing the JSON log\n- **Install-time scaffold removed** — `synthadoc install` no longer runs the LLM scaffold call during installation; scaffold is a separate step (`synthadoc scaffold`) that the user runs after setting their API key, avoiding confusing API-key-not-set errors at install time\n- **AquaFlow LLM evaluation** — 15-question M&A due-diligence benchmark across five models (MiniMax-Think M3, Claude Opus 4.8, Claude Sonnet 4.6, DeepSeek-R1, Qwen Plus); full report at `docs/example/aquaflow/evaluation/report/llm-query-benchmark.md`\n\n### v1.0.0\n\n- **Pre-LLM source sanitizer** — every source document passes through a sanitizer before any LLM call; removes six categories of potentially harmful content (zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars, instruction-override phrases); a `WARN` log entry is emitted when bidi overrides or instruction-override phrases are stripped; zero LLM cost; applied before truncation so the character budget is consumed by clean text only\n- **Per-source truncation flag** — default character limit raised from 8,000 to 32,000 characters per source (~8,000 tokens); truncated sources are flagged with `truncated: true` in the page's `sources:` frontmatter; `--max-source-chars N` CLI flag and matching `POST /jobs/ingest` body field and MCP `synthadoc_ingest` parameter override the limit per run; `synthadoc lint` emits a warning for any page with a truncated source and suggests the override command; configurable via `[ingest] max_source_chars` in `config.toml`\n- **Proportional context budget** — query context is allocated proportionally to the configured model's context window; four configurable slices: `context_wiki_pct` (60%), `context_history_pct` (20%), `context_system_pct` (15%), `context_index_pct` (5%); built-in prefix-matched model context window table covers Claude 4, GPT-4o/turbo/3.5, and GPT-4; unknown models fall back to 128,000 tokens; `context_window = N` in `[query]` overrides the table for local or custom models\n- **Knowledge graph** — wikilink graph computed across all active and draft pages during every lint run; stored in `graph_nodes` and `graph_edges` tables in `audit.db`; edge weight = number of `[[wikilink]]` references between two pages; cluster IDs assigned by Louvain community detection; `GET /graph` REST endpoint returns nodes with per-node enrichment (title, type, state, cluster) and edges; first call after upgrade triggers a background build and returns `{\"status\":\"computing\"}`; web UI **Graph** tab (D3.js force-directed graph) — nodes colored by cluster, click to see page details and an \"Ask about this →\" button that opens a pre-filled chat query\n- **Citation accuracy & rendering improvements** — six correctness fixes to the ingest citation pipeline: (1) full source text passed to the decision LLM so citations can reference any line in the document; (2) `_annotate_citations` bugs A–F resolved (false-positive matches, off-by-one line ranges, duplicate markers, escaped bracket handling, multi-paragraph span errors, and source filename normalisation); (3) `_has_citations` check added — lint warns and suggests a model upgrade when a page has zero citation markers after synthesis; (4) `Key Data` section extraction hardened so numerical facts, rates, and formulas are preserved verbatim across the annotation pass; (5) `obsidianCitationsToGfm` rendering fixed in the web UI — citation superscripts now display correctly in the chat panel alongside streamed answers; (6) reading view set as the default Obsidian display mode for wiki pages\n\n### v0.9.3\n\n- **Backup now includes `hooks/` and `.synthadoc/extracted/`** — both directories were silently omitted from backups. `hooks/` contains user-customised hook scripts; `.synthadoc/extracted/` contains per-source text sidecars and PDF pagemaps written by the ingest agent. Without these, the Source Viewer and Provenance citation modal showed \"Could not read source file\" after restore.\n- **PyPI publish workflow** — `publish.yml` triggers on GitHub Release, builds the wheel with hatchling, and uploads to PyPI via OIDC trusted publishing (no API token stored). `pip install synthadoc` now works.\n- **Web UI bundled in pip package** — `synthadoc/data/web-ui/dist/` is now included in the wheel. `synthadoc web` previously returned 503 on pip-installed instances because the React dist was not packaged.\n- **CI sync checks** — PRs now fail if `synthadoc/data/obsidian-plugin/`, `synthadoc/data/web-ui/`, or `README-pypi.md` are out of sync with their source. All sync-check steps use `shell: bash` for Windows runner compatibility.\n- **`bump_version.py` extended** — now also updates `synthadoc/data/obsidian-plugin/manifest.json` alongside the source manifest, preventing version drift between the bundled copy and the plugin source.\n\n### v0.9.2\n\n- **`synthadoc/data/web-ui/dist/` packaged** — web UI dist files committed to the repo and bundled in the wheel via hatchling artifacts.\n- **`scripts/sync_web_ui.py`** — helper script to sync `web-ui/dist/` into `synthadoc/data/web-ui/dist/` after `npm run build`.\n\n### v0.9.1\n\n- **PyPI metadata** — added `readme`, `keywords`, `classifiers`, and `[project.urls]` to `pyproject.toml` for correct PyPI rendering.\n- **`README-pypi.md`** — generated from `README.md` by `scripts/prepare_pypi_readme.py`; strips `pypi-strip` blocks and rewrites relative links to absolute GitHub URLs.\n- **Branch protection compatibility** — removed post-merge `build-plugin` CI job that pushed compiled plugin directly to `main`. Plugin sync is now enforced in PRs: CI fails if `synthadoc/data/obsidian-plugin/` is out of sync.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 5). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n"}, "0d9cafd563b11c1e599dbc5419960386d4cc1ba3": {"t": 1785767253.216731, "n": 216730, "h": "96c1917ff5d2e593", "c": "# Synthadoc — Design Document\n\n**Version:** 1.1.0 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n29. [Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer)\n30. [Per-Source Truncation Flag](#30-per-source-truncation-flag)\n31. [Proportional Context Budget](#31-proportional-context-budget)\n32. [Knowledge Graph](#32-knowledge-graph)\n33. [Multi-Platform Agent Skill Files](#33-multi-platform-agent-skill-files)\n34. [Synthadoc Scaffold Marker](#34-synthadoc-scaffold-marker)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for Codex CLI, OpenCode, and generic OpenAI agents\n CLAUDE.md ← same content as AGENTS.md; loaded automatically by Claude Code\n GEMINI.md ← same content as AGENTS.md; loaded automatically by Gemini CLI\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the full source text (bounded by `max_source_chars`) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `flag` / `skip`). Pages with `status='active'` are protected — sources that conflict with them trigger `flag` rather than `update` (RULE 1b).\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Orchestrator calls `_guard_ingest_cost()` on the returned cost: soft warn → `logger.warning` only; hard gate → `cost_gate_exceeded` audit event + `fail_permanent` (job → DEAD). On success, job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 0b — Key Data Extraction | None (regex) | Deterministic pre-processor: extracts numbers, percentages, currency amounts, ratios, formulas, underscore identifiers, and date ranges from source text via regex; appends them as a `[Key Data — extracted by pre-processor]` section before any LLM sees the text. Zero token cost; cannot hallucinate. Anchors the synthesis LLM to all quantitative candidates. |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads the full source text (bounded by `max_source_chars`) + BM25 candidates + `purpose.md` scope + status of each candidate page. Outputs per-page action: `create`, `update`, `flag`, `skip`. **RULE 1b:** a page with `status='active'` is human-reviewed and authoritative — when the source provides a conflicting value, date, formula, or conclusion, the action must be `flag`, not `update`. The only exception is a source that adds a topic the page does not yet mention at all. Decision cache version: `v2`. |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. If no citation markers are present in the returned body, a `citation_pass4_no_markers` WARNING audit event is written — the page is still saved but the condition is flagged for re-ingest with a more capable model. Zero-citation results are not cached so a subsequent re-ingest can succeed. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic. **Suppressed when TF fallback was used** (a small corpus with broad coverage may legitimately return fewer than 3 candidates — that is expected, not a gap).\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n| Citation presence (Check 5b) | Page body has ≥ 50 words and zero `^[filename:L-L]` citation markers → WARNING. Diagnostic only — does not block promotion to `active`. Indicates the annotation pass failed to produce markers, usually a model-compatibility issue (use Gemini 2.5 Flash or higher for reliable citation annotation). |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n### ActionAgent\n\nDispatches action-intent queries from the chat UI (e.g. \"activate a draft page\", \"show lint report\", \"archive a stale page\", \"run scaffold\"). Parses the user's free-text intent into a structured action via an LLM extraction prompt, then executes the action against the wiki and returns a human-readable result. When the intent is ambiguous (e.g. \"activate a page\" without specifying which), sets `needs_clarification=True` and returns a prompt and candidate list for the web UI to render as chip buttons.\n\n### ScaffoldAgent\n\nGenerates and updates the wiki's structural pages. Reads all current wiki pages and asks the LLM to organise them into 5–8 domain-appropriate categories, then writes `wiki/index.md` (category index with wikilinks), `wiki/purpose.md` (scope statement), and `AGENTS.md` (ingest and query guidelines). Also stamps a `categories:` field on each page's YAML frontmatter. When `ROUTING.md` already exists, regenerates it to stay in sync with the updated index structure.\n\n### RewriteAgent\n\nRewrites follow-up questions in multi-turn conversations into self-contained, standalone form before BM25 retrieval. Converts context-dependent phrases (\"What came after that?\", \"Tell me more about his early life\") into explicit questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the correct pages without relying on conversation context. Only invoked when conversation history is non-empty. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### SummarizeAgent\n\nCompresses the oldest conversation turns into a single `[Session summary: …]` assistant turn when a session exceeds `conversation_history_turns`. Prevents unbounded context growth across long sessions. Emits a `notice` SSE event the first time compression occurs so the user can see that earlier context was condensed. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### ContextAgent\n\nBuilds token-budgeted context packs for MCP tool consumers. See [Section 20 — Context Packs](#20-context-packs) for full detail.\n\n### SearchDecomposeAgent\n\nDecomposes a web search intent into 1–4 terse keyword search strings optimised for authoritative source retrieval. Uses a separate prompt from `QueryAgent`'s question decomposition — search decomposition asks \"what distinct search strings would find the best sources?\" (terse keyword phrases) rather than \"what distinct questions does this ask?\" (natural-language sub-questions). See [Web search fan-out](#web-search-fan-out) in the IngestAgent section.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n| `session` | `.jsonl` | `claude session`, `codex session`, `cursor session`, `ai session`, `session history` | Extracts human-readable turns from AI coding session transcripts. Supports Claude Code JSONL format and Codex/Cursor format. Filters tool calls, thinking blocks, and sub-agent scaffolding. Short turns (< 20 assistant words, < 3 user words) are skipped. No external dependencies. |\n\n### Session Skill — AI Session History Ingestion\n\nThe `session` skill turns AI coding session history files (`.jsonl`) into searchable wiki pages capturing the problem-solving knowledge exchanged during the session.\n\n**Supported formats**\n\n| Format | File origin | Detection |\n|--------|-------------|-----------|\n| Claude Code | `~/.claude/projects//.jsonl` | `obj.type` ∈ `{\"user\", \"assistant\"}` with nested `obj.message` |\n| Codex / Cursor | Export from OpenAI Codex or Cursor IDE | `obj.role` at top level, no `message` wrapper |\n\nFormat is auto-detected from the first 30 parseable lines. Files that match neither format are parsed as Codex (fallback).\n\n**Filtering rules**\n\n| Content | Kept? | Reason |\n|---------|-------|--------|\n| User text ≥ 3 words | ✓ | Substantive question or instruction |\n| Assistant text ≥ 20 words | ✓ | Substantive answer |\n| User text < 3 words (`\"ok\"`, `\"yes\"`) | ✗ | Too terse to be useful |\n| Assistant text < 20 words | ✗ | Acknowledgement or one-liner |\n| `isSidechain: true` lines | ✗ | Sub-agent scaffolding (not the user's conversation) |\n| `tool_use` blocks | ✗ | Shell commands / file writes — not final output |\n| `tool_result` blocks | ✗ | Raw output — avoids leaking file contents or credentials |\n| `thinking` blocks | ✗ | Internal reasoning — not the final answer |\n| `permission-mode`, `system`, `last-prompt` lines | ✗ | Session metadata, not conversation |\n\nAfter extraction the text passes through Synthadoc's standard pre-LLM source sanitizer\n(zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars,\ninstruction-override phrases) — the same step applied to every PDF, DOCX, URL, and other source type.\nSee [§29 Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer).\n\n**Output format**\n\nEach kept turn is labelled and separated by `---`:\n\n```\n[USER]\nHow do I implement a sliding window algorithm in Python?\n\n---\n\n[ASSISTANT]\nA sliding window algorithm maintains a contiguous subarray by advancing two\npointers simultaneously…\n```\n\n**`suggested_slug`**\n\nThe skill sets `metadata[\"suggested_slug\"]` to `session-YYYY-MM-DD-` where the date comes from the file's `mtime` and the topic from the first substantive user turn (first 6 words, slugified). Example: `session-2026-07-15-how-do-i-implement-sliding`.\n\n**Limitations**\n\n- No chunking in v1.1 — very long sessions are truncated at `max_source_chars` (default 400 000 chars). Split large archives into individual `.jsonl` files.\n- Tool output is excluded by design — re-ingest the original source files if you need the file contents in the wiki.\n- Re-ingesting the same `.jsonl` file uses Synthadoc's standard source-hash dedup — no duplicate pages are created.\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n**Compound identifier expansion:** When a token contains underscores, the tokenizer emits both the compound form and each component as separate tokens (`capex_growth` → `[\"capex_growth\", \"capex\", \"growth\"]`). This expansion is applied at both index time and query time so that human-friendly queries match programmatic identifiers and vice versa. Leading/trailing underscores are stripped before splitting; no empty strings are emitted.\n\n**TF fallback for small corpora:** When all BM25 scores in a result set are zero or negative (IDF collapse on a corpus of ≤ 2 documents, or a term present in every page), the search system falls back to normalised term frequency (TF) scoring: `TF(doc, query) = Σcount(term, doc) / len(doc_tokens)`. Results are returned in the same format as BM25 results with a `tf_fallback: True` flag on each `SearchResult`. This field is recorded in the audit trail and consumed by gap detection (see below).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp, cascade_links_removed_from: [str]}` _(cascade field added v1.0.2)_ |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n| `GET` | `/graph` _(v1.0.0)_ | — | `{status, node_count, edge_count, cluster_count, nodes: [...], edges: [...]}` or `{status: \"computing\"}` on first call |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n**External file paths** (outside the wiki root) are supported when the request comes from\n`127.0.0.1` or `::1` and the payload includes `\"allow_external_paths\": true`. The CLI sets\nthis flag automatically for all local file sources. Remote clients cannot set this flag\n(the server ignores it for non-localhost connections) to prevent arbitrary file reads\nin server-exposed deployments.\n\nThis is required for ingesting files that live outside the wiki directory by design —\nfor example, Claude Code session transcripts at `~/.claude/projects//.jsonl`.\n\n**Client behaviour summary:**\n\n| Client | Sends `allow_external_paths` | Can ingest outside wiki root? |\n|--------|------------------------------|-------------------------------|\n| CLI (`synthadoc ingest `) | `true` (automatic, local file paths only) | Yes — server is localhost |\n| Obsidian plugin | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Web UI | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Direct HTTP API (localhost) | Set manually in request body | Yes — if request from 127.0.0.1/::1 |\n| Direct HTTP API (remote) | Ignored by server | No — server silently treats as `false` |\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 1.1.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n**Reading View default:** `synthadoc plugin install` and `synthadoc plugin upgrade` write `\"defaultViewMode\": \"preview\"` to `.obsidian/app.json` after copying the plugin files. This causes Obsidian to open new notes in Reading View by default, which is required for `^[filename:L-L]` citation chips to render. Only `defaultViewMode` is written; all other keys in `app.json` are preserved. If `app.json` is malformed JSON the write is skipped and a warning is logged — the install still succeeds.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports two things: dangling slugs (pages listed in ROUTING.md that no longer exist in the wiki) and unassigned slugs (pages that exist in index.md but are not listed in any ROUTING.md branch). Enabled only when ROUTING.md exists. **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n| `Graph: show knowledge graph` _(v1.1.0)_ | Open the knowledge graph panel — draggable modal with Canvas force-directed graph, type filter dropdown, cluster legend, hover tooltip showing title/slug/type/state/cluster/connections, and click-to-open page in current pane. System pages (index, overview, dashboard, purpose, log) are excluded. Capped at 300 most-connected nodes for large wikis with an explanatory banner. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [-w wiki] — generate ROUTING.md from index.md branch structure\n│ ├── validate [-w wiki] — report dangling slugs and cross-branch duplicates\n│ └── clean [-w wiki] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [-w wiki]\n├── candidates\n│ ├── list [-w wiki]\n│ ├── promote |--all [-w wiki]\n│ └── discard |--all [-w wiki]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n# Citation pass (Pass 4) tuning — these two settings work together:\n# citation_source_lines — how many lines of the source the LLM sees when placing ^[...] markers.\n# Increase if lint reports out_of_range on long sources (transcripts, PDFs).\n# citation_max_tokens — output token budget for the annotated section returned by the LLM.\n# Increase if you raise citation_source_lines and have long wiki sections.\n# citation_source_lines = 400\n# citation_max_tokens = 8192\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Emit `logger.warning` in server log when per-job ingest cost exceeds this threshold; job continues normally |\n| `cost.hard_gate_usd` | float | `2.00` | Permanently fail the ingest job (DEAD) and record a `cost_gate_exceeded` audit event when per-job cost exceeds this threshold |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × (backup_count + 1)` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n| `ingest.max_source_chars` | int | `32000` | Character limit applied to each source before the LLM call. Sources exceeding this limit are truncated; the page's `sources:` frontmatter entry gets `truncated: true` and lint emits a warning. Override per-run with `--max-source-chars N`. _(v1.0.0)_ |\n| `ingest.citation_source_lines` | int | `400` | Number of source lines the LLM sees during Pass 4 (citation annotation). Increase if lint reports `out_of_range` on long sources such as transcripts or PDFs. Raise `citation_max_tokens` in proportion when increasing this value. |\n| `ingest.citation_max_tokens` | int | `8192` | Output token budget for the annotated section returned by Pass 4. Increase when `citation_source_lines` is raised or wiki sections are long, to avoid truncated annotations. |\n| `query.context_wiki_pct` | float | `0.60` | Fraction of the model context window reserved for wiki source pages. _(v1.0.0)_ |\n| `query.context_history_pct` | float | `0.20` | Fraction reserved for conversation history. _(v1.0.0)_ |\n| `query.context_system_pct` | float | `0.15` | Fraction reserved for system prompt and instructions. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_index_pct` | float | `0.05` | Fraction reserved for the wiki index preamble. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_window` | int | _(auto)_ | Override the built-in context window lookup for the configured model. Use when running a local or custom model whose window size is not in the built-in table. _(v1.0.0)_ |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement is wired into the ingest pipeline: `Orchestrator._guard_ingest_cost()` is called after each ingest LLM call with the returned cost, before the job is marked complete.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | `logger.warning` in server log; job continues normally |\n| `hard_gate_usd` | $2.00 | Server/ingest path: records `cost_gate_exceeded` audit event + permanently fails job (DEAD). Interactive CLI: prompts `Proceed? [y/N]` |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` by default. The bind address is configurable via `server.host` in `config.toml` (e.g. `host = \"0.0.0.0\"` to expose on all interfaces for LAN access). No built-in authentication — restrict via firewall when exposing beyond loopback. Remote access without a reverse proxy is not recommended on shared or networked machines.\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs and unassigned slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `-w ` / `--wiki ` to target a specific wiki.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; when all local source files are missing from disk the page is transitioned to `archived` (if only some sources are missing it is transitioned to `stale` instead). Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: lint evaluates all local (non-URL) source paths listed in the page's frontmatter. If **all** of them are missing from disk, the page is transitioned to `archived`. If only **some** are missing — meaning valid content from at least one source still exists — the page is transitioned to `stale` instead, preserving it for re-ingest rather than discarding it. _(v1.0.2: previously any single missing source triggered an immediate archive.)_\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Cascade link cleanup\n\nWhenever a page transitions to `archived` — whether triggered manually (`synthadoc lifecycle archive`), via the Obsidian Lifecycle modal, the MCP `synthadoc_lifecycle` tool, or automatically by lint detecting a missing source — Synthadoc immediately scans all other active pages and removes every `[[archived-slug]]` wikilink pointing to the now-archived page.\n\n- **Inline links** — `[[archived-slug]]` becomes plain text (the display text if an alias was used, otherwise the slug itself).\n- **List-item links** — a list item whose only content is `[[archived-slug]]` is dropped entirely.\n- **Archived pages are skipped** — pages already archived are not rewritten.\n- **System pages are skipped** — index, overview, dashboard, log, and purpose pages are excluded.\n\nThe cleanup runs synchronously before the response is returned, so the wiki is consistent immediately. The CLI and MCP responses include a `cascade_links_removed_from` field listing the affected slugs. During `lint run`, cascade is batched once after all auto-archive transitions complete, and the count is added to `dangling_links_removed` in the lint report.\n\nLint's existing periodic dangling-link cleanup (`lint run --scope orphans`) remains in place as a safety net for any dead links that predate this feature or arrive through other paths.\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 5) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 5 # turns to include in each request (default: 5; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp, cascade_links_removed_from: [str]}` or `{error, cascade_links_removed_from: []}` _(cascade field added v1.0.2)_ | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions for Codex CLI / OpenCode (if present)\n├── CLAUDE.md ← same content; loaded by Claude Code (if present)\n├── GEMINI.md ← same content; loaded by Gemini CLI (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── hooks/ ← user hook scripts (always included)\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ ├── extracted/ ← text sidecars + PDF pagemaps (always included)\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `logs/`.\n\n> **v0.9.3 fix:** `hooks/` and `.synthadoc/extracted/` were previously missing from backups. Omitting `extracted/` caused the Source Viewer and Provenance citation modal to show \"Could not read source file\" after restore.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## 29. Pre-LLM Source Sanitizer\n\nEvery source document passes through a sanitizer immediately after text extraction, before any LLM call. The sanitizer removes six categories of content that could manipulate the LLM or degrade compilation quality:\n\n| Category | Action | Warning logged? |\n|---|---|---|\n| Zero-width characters (U+200B, U+200C, U+200D, U+FEFF) | Removed silently | No |\n| Bidi override characters (U+202A–U+202E, U+2066–U+2069) | Removed | Yes |\n| HTML comments (``) | Removed silently | No |\n| Hidden CSS spans (`display:none`, `visibility:hidden`) | Removed silently | No |\n| Base64 blobs ≥ 200 consecutive characters | Replaced with `[base64 content removed]` | Yes |\n| Instruction-override phrases (8 patterns: \"ignore previous instructions\", \"disregard the above\", \"override your system prompt\", etc.) | Replaced with `[redacted]` | Always |\n\nWhen content is removed and a warning is appropriate, the server logs a `WARN` entry:\n```\n[WARN] sanitizer stripped content from 'papers/survey.pdf': bidi overrides, instruction-override phrase\n```\n\nThis is a pure pre-processing step with zero LLM cost. Truncation (if configured) is applied after sanitization, so the character budget is consumed by clean text only.\n\n---\n\n## 30. Per-Source Truncation Flag\n\nLarge source documents are truncated before they reach the LLM. By default the limit is **32,000 characters** (~8,000 tokens), raised from the prior hardcoded 8,000-character limit. When a source exceeds this limit, its compiled wiki page records a `truncated: true` flag in its `sources:` frontmatter entry.\n\n### Configuration\n\n```toml\n[ingest]\nmax_source_chars = 32000 # default; raise for large PDFs or books\n```\n\n### Per-ingest override\n\nThe limit can be raised for a single ingest run without modifying the config:\n\n```bash\nsynthadoc ingest papers/large-textbook.pdf --max-source-chars 128000\n```\n\nThe same `max_source_chars` field is accepted by the HTTP `POST /jobs/ingest` body and the MCP `synthadoc_ingest` tool.\n\n### Lint surfacing\n\n`synthadoc lint` emits a warning for any page that has a truncated source:\n\n```\n[WARN] quantum-computing.md: source 'papers/large-survey.pdf' was truncated at ingest\n (source exceeded max_source_chars=32000 — 87,412 chars in source).\n To re-ingest with a higher limit (this source only):\n synthadoc ingest papers/large-survey.pdf --max-source-chars 128000\n To raise the limit for all future ingests:\n set [ingest] max_source_chars = 128000 in your config\n```\n\n---\n\n## 31. Proportional Context Budget\n\nQuery context is allocated proportionally to the configured model's context window, replacing the prior hard cap. The default allocation:\n\n| Slice | Default | Purpose |\n|---|---|---|\n| `context_wiki_pct` | 60% | Wiki page content (ranked by BM25 + vector score) |\n| `context_history_pct` | 20% | Chat turn history (newest-first, oldest dropped when over budget) |\n| `context_system_pct` | 15% | System prompt and purpose document |\n| `context_index_pct` | 5% | `ROUTING.md` and search index |\n\nPercentages must sum to ≤ 100. Configure in `[query]`:\n\n```toml\n[query]\ncontext_window = 0 # 0 = auto-detect from known model map\ncontext_wiki_pct = 60\ncontext_history_pct = 20\ncontext_system_pct = 15\ncontext_index_pct = 5\n```\n\n> **Note:** `context_system_pct` and `context_index_pct` are parsed and validated but not yet enforced as hard caps in this release — the system prompt uses a conservative fixed limit, which is well within the configured 15% slice for all supported models. Full per-slice enforcement is planned for v1.1.\n\n### Model context window map\n\nSynthadoc ships a built-in prefix-matched table:\n\n| Model | Context window |\n|---|---|\n| `claude-opus-4*`, `claude-sonnet-4*`, `claude-haiku-4*` | 200,000 tokens |\n| `gpt-4o*`, `gpt-4-turbo*` | 128,000 tokens |\n| `gpt-4` (exact) | 8,192 tokens |\n| `gpt-3.5-turbo*` | 16,385 tokens |\n| Unknown / fallback | 128,000 tokens |\n\nSet `context_window = N` in config to override the map (useful for local Ollama models with non-standard context sizes).\n\nAt the defaults on a 200k-token model: wiki slice ≈ 480,000 chars — far more than any practical query needs. On constrained models, the budget scales down automatically so the same config works across model families.\n\n---\n\n## 32. Knowledge Graph\n\nSynthadoc computes a wikilink graph across all active and draft pages during every lint run. The graph is stored in two tables in `audit.db`:\n\n```sql\nCREATE TABLE graph_nodes (\n slug TEXT PRIMARY KEY,\n cluster_id INTEGER NOT NULL,\n updated_at TEXT NOT NULL\n);\n\nCREATE TABLE graph_edges (\n from_slug TEXT NOT NULL,\n to_slug TEXT NOT NULL,\n weight INTEGER NOT NULL DEFAULT 1,\n edge_type TEXT NOT NULL DEFAULT 'mixed',\n PRIMARY KEY (from_slug, to_slug)\n);\n```\n\nEdge weight is a composite signal from two sources:\n\n| Edge type | Source | Weight contribution |\n|-----------|--------|---------------------|\n| `wikilink` | `[[slug]]` occurrences in page body | +1 per occurrence |\n| `co_source` | pages compiled from the same source file (matched by hash) | +2 per shared source |\n\nMost edges are `mixed` (have both types). Pure `co_source` edges surface hidden relationships — pages compiled from the same source document are linked immediately after ingest, before any wikilinks are created. `edge_type` is stored on each edge so the web UI can render co-source edges with a different visual style.\n\nCluster IDs are assigned by Louvain community detection — pages that link densely to each other end up in the same cluster.\n\n### REST API\n\n**`GET /graph`** — returns the full graph with per-node enrichment:\n\n```json\n{\n \"status\": \"ready\",\n \"node_count\": 45,\n \"edge_count\": 123,\n \"cluster_count\": 4,\n \"nodes\": [\n {\"slug\": \"quantum-error-correction\", \"title\": \"Quantum Error Correction\",\n \"type\": \"concept\", \"state\": \"active\", \"cluster_id\": 2}\n ],\n \"edges\": [\n {\"from\": \"quantum-error-correction\", \"to\": \"shor-algorithm\", \"weight\": 3, \"edge_type\": \"mixed\"}\n ]\n}\n```\n\nWhen the graph tables are empty (first call after upgrade, or before the first lint run), the server returns `{\"status\": \"computing\"}` and starts a background build immediately. The web UI polls every 2 seconds and renders the graph when the status becomes `\"ready\"`.\n\n### Web UI\n\nThe **Graph** tab (alongside Chat in the top nav) renders a D3.js force-directed graph. Nodes are colored by cluster. Edge thickness is proportional to weight (1–4 px). Pure `co_source` edges (no wikilink) are rendered with a dashed stroke, making hidden source-based relationships visually distinct from explicit cross-links. Clicking a node opens a sidebar showing the page title, type and lifecycle state badges, cluster assignment, and an **\"Ask about this →\"** button that switches to the Chat tab with a pre-filled query.\n\nControls: zoom in/out (scroll or pinch), drag nodes, filter by page type.\n\n### System page exclusion _(v1.1.0)_\n\nThe five structural system pages — `overview`, `index`, `dashboard`, `purpose`, and `log` — are excluded from both `graph_nodes` and `graph_edges`. These pages link to nearly every other page and would act as false hubs, inflating edge counts and distorting Louvain cluster detection. System page exclusion is applied inside `_build_graph()` before any edge computation.\n\n### Obsidian knowledge graph panel _(v1.1.0)_\n\n→ Full description: [§33 Multi-Platform Agent Skill Files](#33-multi-platform-agent-skill-files) covers scaffold; the graph panel itself is documented in [§8 Obsidian Plugin](#8-obsidian-plugin) command table.\n\nThe Obsidian plugin includes a **draggable modal** knowledge graph panel (`Graph: show knowledge graph` command). It fetches from the same `GET /graph` endpoint as the web UI and renders a force-directed Canvas 2D simulation using Verlet integration (no external library). System pages are already excluded from the server response.\n\n| Feature | Obsidian panel | Web UI Graph tab |\n|---------|----------------|-----------------|\n| Engine | Canvas 2D + Verlet | D3.js SVG |\n| Node click | Opens the wiki page in the current pane | Opens the node sidebar with \"Ask about this →\" |\n| **Best for** | **Navigating to and editing pages** | **Asking questions about a page** |\n| Draggable panel | Yes — drag header to reposition | Fixed tab layout |\n| Node cap | 300 most-connected (see note) | No cap |\n| Cluster legend | Yes | Yes |\n| Type filter | Yes | Yes |\n\n**Rule of thumb:** Use the **web UI** when you want to ask questions about a node. Use the **Obsidian panel** when you want to navigate to and edit a page.\n\n**Node cap rationale:** The Obsidian panel uses a hand-written Verlet integrator with O(n²) pairwise charge repulsion — every node repels every other node on each animation tick. At 300 nodes that is ~90,000 operations per frame, which runs smoothly at 60 fps inside Obsidian's Electron renderer. At 500 nodes it is ~250,000 operations per frame and starts to lag noticeably. The web UI has no cap because D3's `forceManyBody` uses a Barnes-Hut tree approximation (O(n log n)), which scales to thousands of nodes without frame-rate issues. When the cap is hit, nodes are ranked by degree and the 300 most-connected are shown; a banner reports the total page count and the number of hidden nodes, and suggests using the type filter to bring the set below 300.\n\n→ User walkthrough: [Quick-Start Guide §24 — Knowledge Graph](docs/user-quick-start-guide.md#knowledge-graph)\n\n---\n\n## 33. Multi-Platform Agent Skill Files\n\n\n\n`synthadoc init` and `synthadoc scaffold` write three agent skill files at the wiki root:\n\n| File | Format | Read by |\n|------|--------|---------|\n| `AGENTS.md` | OpenAI Agents SDK / Codex / OpenCode | Codex CLI, OpenCode, generic agents |\n| `CLAUDE.md` | Claude Code / Anthropic | Claude Code (loaded automatically when the wiki folder is opened) |\n| `GEMINI.md` | Gemini CLI | Gemini CLI |\n\nAll three files carry **identical content**: a title block, the LLM-generated domain guidelines for the wiki, a quick-reference CLI command table, and an MCP tool table. The only difference is the heading comment. This means any AI coding tool that opens the wiki root gets full Synthadoc guidance without manual setup.\n\n### Content generated by `scaffold`\n\nEach skill file is generated from the same LLM call that produces `purpose.md` and `index.md`. The LLM receives the current wiki page list and domain description, then generates:\n\n1. **Domain label** — a precise human-readable name inferred from wiki content (e.g. `History of Computing`, not `My Wiki`)\n2. **Domain guidelines** — 3 domain-specific ingest and query guidelines as plain sentences\n3. **Quick Reference table** — pre-filled with the current wiki name and port\n\nIf the wiki was created before v1.1.0, running `synthadoc scaffold` once regenerates all three files and adds `CLAUDE.md` and `GEMINI.md` to any wiki that only had `AGENTS.md`.\n\n### Template source\n\nThe instruction body is defined in `synthadoc/cli/_init.py` as `_AGENT_INSTRUCTION_BODY` and shared by all three templates, keeping the files permanently in sync. The scaffold agent substitutes `{domain}`, `{guidelines}`, and `{port}` at write time.\n\n---\n\n## 34. Synthadoc Scaffold Marker\n\n\n\nThe scaffold marker `` (written as an HTML comment) separates **user-authored content** (preserved) from **scaffold-managed content** (rewritten on each `synthadoc scaffold` run).\n\n### Single-marker mode (index.md)\n\nThe marker appears once, below the H1 heading in `index.md`. Everything above the marker is the user zone; everything below is replaced with new LLM-generated categories.\n\n```markdown\nMy custom introduction — maintained by hand.\n\n\n\n## Pioneers and Visionaries\n- [[alan-turing]]\n...\n```\n\nIf the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### Per-section multi-marker mode (purpose.md)\n\n`purpose.md` uses a **per-section** approach: the marker appears inside each `## Section` heading. Within a section, content the user writes **above** the marker is preserved; content below is replaced with fresh LLM output. Sections without a marker in the existing file are replaced entirely (and a marker is added). Sections in the existing file that the scaffold template does not include are kept unchanged.\n\n```markdown\n## Overview\n\nMy custom overview note.\n\n\n\nLLM-generated overview content goes here.\n\n## What Belongs\n\n\n\nLLM-generated scope definition.\n```\n\nThis design lets users annotate any section of `purpose.md` without losing their notes on the next scaffold run.\n\n### Marker preservation guarantee\n\n- `preserve_user_zone()` in `scaffold_agent.py` handles both modes.\n- If a section has no marker in the existing file, `preserve_user_zone()` treats the entire section body as user content and preserves it, replacing only what falls below any marker it finds.\n- The scaffold NEVER touches `config.toml`, `dashboard.md`, or any wiki page (`wiki/*.md`).\n\n→ User walkthrough: [Quick-Start Guide §14 — Enrich the wiki with scaffold](docs/user-quick-start-guide.md#scaffold)\n\n---\n\n## Customization\n\nSynthadoc exposes four extension points — all hot-loaded, no server restart required.\n\n### Custom skills (new file formats)\n\nSubclass `BaseSkill` (Apache-2.0 — no AGPL obligation on your skill code), drop the folder in `/skills/` or `~/.synthadoc/skills/`, and Synthadoc picks it up on the next ingest. Skills match by file extension or intent prefix and support any Unicode text, including CJK prefixes.\n\n→ Full interface, manifest format, and examples: [§5 Skills System](#5-skills-system) and [§17 Plugin Development Guide](#17-plugin-development-guide)\n\n### Custom LLM providers\n\nSubclass `LLMProvider` from `synthadoc/providers/base.py` (Apache-2.0) and place the file in `~/.synthadoc/providers/` or the wiki `providers/` directory. Switch active provider with one config line.\n\n→ Full interface and wiring instructions: [§10 Configuration — Provider switching](#10-configuration)\n\n### Hooks\n\nShell scripts (any language) that fire on `on_ingest_complete` and `on_lint_complete`. Receive a JSON context on stdin. Set `blocking = true` to gate the operation on the hook's exit code — useful for CI pipelines and quality gates.\n\n→ Full event schema, context JSON, and blocking behaviour: [§11 Hook System](#11-hook-system)\n\n### Cache control\n\nThree cache layers (embedding, LLM response, provider prompt cache) invalidate automatically on source-file change (SHA-256). Force a fresh call with `--force`, or wipe all cached responses with `synthadoc cache clear -w `.\n\n→ Layer-by-layer breakdown and invalidation rules: [§12 Cache System](#12-cache-system)\n\n### Multi-Platform Agent Skill Files\n\nEvery Synthadoc wiki ships three companion files that give AI coding agents a complete, self-contained operating guide for the wiki:\n\n| File | Platform |\n|---|---|\n| `AGENTS.md` | Codex CLI, OpenCode, and any agent that reads the AGENTS.md convention |\n| `CLAUDE.md` | Claude Code (this session's tool) — highest-priority instruction source in Claude's hierarchy |\n| `GEMINI.md` | Gemini CLI — relevant because Synthadoc's default LLM provider is `gemini-2.5-flash-lite` |\n\nAll three files share identical body content generated from the same template; they differ only in their H1 heading so each agent recognises its own file first. The body covers:\n\n- **Domain guidelines** — LLM-generated bullet list of domain-specific ingest and query rules (produced by `scaffold`)\n- **Quick reference table** — every common CLI command with the correct `synthadoc` syntax\n- **Server startup** — how to start the server and verify it is running\n- **Ingest examples** — local file, URL, YouTube, agent session `.jsonl`, force reingest, dry-run\n- **Query** — basic and streaming query syntax; citation marker format\n- **Lint** — when to run and what each check does\n- **Lifecycle** — transition commands with state names\n- **Page schema** — frontmatter structure with all fields\n- **MCP tools table** — all 12 tools with purpose descriptions\n\n**How the files are created and updated:**\n\n1. `synthadoc init` — writes all three files with default domain guidelines and the wiki's configured port.\n2. `synthadoc scaffold` — regenerates all three with LLM-produced domain guidelines derived from the current wiki state. The `scaffold` command prints a confirmation line for each file.\n\n**Use cases:**\n\n- Open Claude Code in a Synthadoc wiki directory and it immediately knows how to ingest, query, lint, and manage lifecycle — no additional setup. Claude Code can also start the server in background mode (`synthadoc serve -w -b`), stop it cleanly, and orchestrate multi-step workflows, effectively acting as a control plane for the wiki.\n- Gemini CLI reads `GEMINI.md` and can run `synthadoc` commands with the correct flags without asking the user for help.\n- CI pipelines with Codex CLI use `AGENTS.md` to drive automated lint and lifecycle transitions on PR merge.\n- Agent session files produced by Claude Code (`.jsonl` under `~/.claude/projects/`) can themselves be ingested into the wiki, capturing the reasoning behind design decisions alongside the source documents.\n\n→ See [README.md — Interfaces & Integration](../README.md#interfaces--integration) for the feature comparison row and quick-start instructions.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v1.1.0\n\n- **Weighted knowledge graph edges** — graph edges now carry two signals: wikilink occurrences (+1 each) and co-source connections — pages compiled from the same source file (matched by SHA-256 hash) — (+2 per shared source). `edge_type` field added to `graph_edges` table (`wikilink`, `co_source`, or `mixed`). `GET /graph` exposes `edge_type` per edge. Web UI renders edge thickness proportional to weight (1–4 px) and dashed lines for pure co-source edges. Co-source edges appear immediately after ingest — before any wikilinks — surfacing hidden relationships in the graph. Schema migrated to version 3.\n- **Multi-platform agent skill files** — `synthadoc init` and `synthadoc scaffold` write `CLAUDE.md`, `AGENTS.md`, and `GEMINI.md` at wiki root; all three carry LLM-generated domain guidelines and the full CLI quick-reference so Claude Code, Codex CLI, OpenCode, Pi, and Gemini CLI users get first-class guidance without manual setup. Run `synthadoc scaffold` once on any pre-v1.1.0 wiki to generate the new files. See [§33](#33-multi-platform-agent-skill-files).\n- **Session history ingestion skill** — `.jsonl` session files from Claude Code and Codex CLI are ingested as wiki pages; human turns and substantive assistant responses extracted; tool calls, thinking blocks, and sub-agent scaffolding skipped; format auto-detected; `suggested_slug` derived from session date and first user message. Converts ephemeral problem-solving knowledge into searchable wiki entries. See [§5 Skills System — Session Skill](#session-skill--ai-session-history-ingestion).\n- **Obsidian knowledge graph panel** — new `Graph: show knowledge graph` command in the Obsidian plugin opens a draggable modal with a Canvas 2D force-directed graph. Features: Verlet integration, type filter dropdown, cluster legend, hover tooltip (title, slug, type, state, cluster, connections), click-to-open page in current pane, node drag to repin. Closes only via the × button — Escape and outside-click do not close the panel, matching the behaviour of other Synthadoc modals. **Large-wiki cap:** when the node set exceeds 300 (after any type filter is applied), nodes are ranked by degree (total edge count) and the 300 most-connected are displayed; edges to capped-out nodes are removed; a banner explains the cap and suggests using the type filter to narrow the view below 300. See [§8 Obsidian Plugin](#8-obsidian-plugin) and [§32 Knowledge Graph — Obsidian panel](#obsidian-knowledge-graph-panel-v110).\n- **Per-section scaffold markers in `purpose.md`** — `purpose.md` now uses per-section `` markers. Users can write content above the marker in any `## Section` block; that content is preserved on every re-scaffold. A single-marker mode for `index.md` continues to work as before. See [§34 Synthadoc Scaffold Marker](#34-synthadoc-scaffold-marker).\n- **System pages excluded from knowledge graph** — `overview`, `index`, `dashboard`, `purpose`, and `log` are filtered out of `_build_graph()` before any edge computation so they cannot act as false hubs in Louvain cluster detection. Both the Obsidian graph panel and the web UI Graph tab reflect this exclusion.\n- **Web UI graph cluster legend** — the web UI Graph tab now shows a cluster legend below the canvas (matching the Obsidian plugin panel), with a color swatch and label for each Louvain cluster present in the filtered node set.\n- **`_merge_section_into_page` deduplication** — force re-ingest of the same source file (e.g. an updated `.jsonl` session) now replaces the page body rather than appending a duplicate section. `_files_match()` normalizes absolute vs. relative paths so the same file is recognized regardless of how it was passed to the ingest command.\n- **`purpose.md` domain label inference** — `scaffold` now infers a precise `domain_label` from current wiki content (e.g. `History of Computing`) instead of using a generic placeholder; the label is used in all three agent skill files and in `purpose.md`'s Overview section.\n- **CI: Windows runner fix** — `scripts/sync_plugin.py` and `scripts/sync_web_ui.py` now use `shutil.copy2` instead of `cp -f`, resolving CI failures on Windows runners for the sync check.\n\n### v1.0.2\n\n- **Cascade link cleanup on archive** — archiving any page immediately removes every `[[slug]]` wikilink pointing to it from all other active pages, without waiting for the next `lint run`. Inline links become plain text; list-item-only links are dropped entirely. The HTTP `POST /lifecycle/transition` and MCP `synthadoc_lifecycle` responses now include a `cascade_links_removed_from` field listing the rewritten page slugs. During `lint run`, cascade runs once after all auto-archive transitions complete and the count is included in the lint report's `dangling_links_removed` field. System pages (index, dashboard, purpose, log, overview) and already-archived pages are skipped. The existing dangling-link lint pass remains as a safety net for dead links that predate this release.\n- **Multi-source partial-archive fix** — when a page has multiple local source files and only some are missing from disk, lint now transitions the page to `stale` (not `archived`), preserving it for re-ingest. The page is only archived when every local source file is gone. URL sources are not affected by this check.\n\n### v1.0.1\n\n- **Scaffold JSON extraction hardened** — replaced the greedy regex used to extract the LLM's JSON response with a brace-balanced parser; eliminates false-positive matches on `[[wikilink]]` syntax that previously caused `json.loads()` failures and silent scaffold corruption\n- **Generated file validation** — scaffold now validates every LLM-generated file (index, purpose, AGENTS.md) for minimum length, required frontmatter, and `[[wikilink]]` presence before writing to disk; validation failure marks the job as failed with a structured error message rather than silently writing a malformed page\n- **ROUTING.md regeneration validation** — `routing init` and scheduled routing regeneration now validate the regenerated file for minimum line count and anchor presence before applying it; prevents a corrupt or truncated ROUTING.md from entering the wiki\n- **Scaffold failure reporting** — scaffold skip and failure reasons are now surfaced in the terminal output and job error field; previously silent failures (e.g. LLM timeout, JSON parse error) are now visible without tailing the JSON log\n- **Install-time scaffold removed** — `synthadoc install` no longer runs the LLM scaffold call during installation; scaffold is a separate step (`synthadoc scaffold`) that the user runs after setting their API key, avoiding confusing API-key-not-set errors at install time\n- **AquaFlow LLM evaluation** — 15-question M&A due-diligence benchmark across five models (MiniMax-Think M3, Claude Opus 4.8, Claude Sonnet 4.6, DeepSeek-R1, Qwen Plus); full report at `docs/example/aquaflow/evaluation/report/llm-query-benchmark.md`\n\n### v1.0.0\n\n- **Pre-LLM source sanitizer** — every source document passes through a sanitizer before any LLM call; removes six categories of potentially harmful content (zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars, instruction-override phrases); a `WARN` log entry is emitted when bidi overrides or instruction-override phrases are stripped; zero LLM cost; applied before truncation so the character budget is consumed by clean text only\n- **Per-source truncation flag** — default character limit raised from 8,000 to 32,000 characters per source (~8,000 tokens); truncated sources are flagged with `truncated: true` in the page's `sources:` frontmatter; `--max-source-chars N` CLI flag and matching `POST /jobs/ingest` body field and MCP `synthadoc_ingest` parameter override the limit per run; `synthadoc lint` emits a warning for any page with a truncated source and suggests the override command; configurable via `[ingest] max_source_chars` in `config.toml`\n- **Proportional context budget** — query context is allocated proportionally to the configured model's context window; four configurable slices: `context_wiki_pct` (60%), `context_history_pct` (20%), `context_system_pct` (15%), `context_index_pct` (5%); built-in prefix-matched model context window table covers Claude 4, GPT-4o/turbo/3.5, and GPT-4; unknown models fall back to 128,000 tokens; `context_window = N` in `[query]` overrides the table for local or custom models\n- **Knowledge graph** — wikilink graph computed across all active and draft pages during every lint run; stored in `graph_nodes` and `graph_edges` tables in `audit.db`; edge weight = number of `[[wikilink]]` references between two pages; cluster IDs assigned by Louvain community detection; `GET /graph` REST endpoint returns nodes with per-node enrichment (title, type, state, cluster) and edges; first call after upgrade triggers a background build and returns `{\"status\":\"computing\"}`; web UI **Graph** tab (D3.js force-directed graph) — nodes colored by cluster, click to see page details and an \"Ask about this →\" button that opens a pre-filled chat query\n- **Citation accuracy & rendering improvements** — six correctness fixes to the ingest citation pipeline: (1) full source text passed to the decision LLM so citations can reference any line in the document; (2) `_annotate_citations` bugs A–F resolved (false-positive matches, off-by-one line ranges, duplicate markers, escaped bracket handling, multi-paragraph span errors, and source filename normalisation); (3) `_has_citations` check added — lint warns and suggests a model upgrade when a page has zero citation markers after synthesis; (4) `Key Data` section extraction hardened so numerical facts, rates, and formulas are preserved verbatim across the annotation pass; (5) `obsidianCitationsToGfm` rendering fixed in the web UI — citation superscripts now display correctly in the chat panel alongside streamed answers; (6) reading view set as the default Obsidian display mode for wiki pages\n\n### v0.9.3\n\n- **Backup now includes `hooks/` and `.synthadoc/extracted/`** — both directories were silently omitted from backups. `hooks/` contains user-customised hook scripts; `.synthadoc/extracted/` contains per-source text sidecars and PDF pagemaps written by the ingest agent. Without these, the Source Viewer and Provenance citation modal showed \"Could not read source file\" after restore.\n- **PyPI publish workflow** — `publish.yml` triggers on GitHub Release, builds the wheel with hatchling, and uploads to PyPI via OIDC trusted publishing (no API token stored). `pip install synthadoc` now works.\n- **Web UI bundled in pip package** — `synthadoc/data/web-ui/dist/` is now included in the wheel. `synthadoc web` previously returned 503 on pip-installed instances because the React dist was not packaged.\n- **CI sync checks** — PRs now fail if `synthadoc/data/obsidian-plugin/`, `synthadoc/data/web-ui/`, or `README-pypi.md` are out of sync with their source. All sync-check steps use `shell: bash` for Windows runner compatibility.\n- **`bump_version.py` extended** — now also updates `synthadoc/data/obsidian-plugin/manifest.json` alongside the source manifest, preventing version drift between the bundled copy and the plugin source.\n\n### v0.9.2\n\n- **`synthadoc/data/web-ui/dist/` packaged** — web UI dist files committed to the repo and bundled in the wheel via hatchling artifacts.\n- **`scripts/sync_web_ui.py`** — helper script to sync `web-ui/dist/` into `synthadoc/data/web-ui/dist/` after `npm run build`.\n\n### v0.9.1\n\n- **PyPI metadata** — added `readme`, `keywords`, `classifiers`, and `[project.urls]` to `pyproject.toml` for correct PyPI rendering.\n- **`README-pypi.md`** — generated from `README.md` by `scripts/prepare_pypi_readme.py`; strips `pypi-strip` blocks and rewrites relative links to absolute GitHub URLs.\n- **Branch protection compatibility** — removed post-merge `build-plugin` CI job that pushed compiled plugin directly to `main`. Plugin sync is now enforced in PRs: CI fails if `synthadoc/data/obsidian-plugin/` is out of sync.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 5). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n"}, "7acf4972d325ba23b9b8351ad790e06fd9efbc2a": {"t": 1785767255.0482426, "n": 206626, "h": "94c419939173bb25", "c": "# Synthadoc — Design Document\n\n**Version:** 1.0.2 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n29. [Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer)\n30. [Per-Source Truncation Flag](#30-per-source-truncation-flag)\n31. [Proportional Context Budget](#31-proportional-context-budget)\n32. [Knowledge Graph](#32-knowledge-graph)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the full source text (bounded by `max_source_chars`) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `flag` / `skip`). Pages with `status='active'` are protected — sources that conflict with them trigger `flag` rather than `update` (RULE 1b).\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Orchestrator calls `_guard_ingest_cost()` on the returned cost: soft warn → `logger.warning` only; hard gate → `cost_gate_exceeded` audit event + `fail_permanent` (job → DEAD). On success, job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 0b — Key Data Extraction | None (regex) | Deterministic pre-processor: extracts numbers, percentages, currency amounts, ratios, formulas, underscore identifiers, and date ranges from source text via regex; appends them as a `[Key Data — extracted by pre-processor]` section before any LLM sees the text. Zero token cost; cannot hallucinate. Anchors the synthesis LLM to all quantitative candidates. |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads the full source text (bounded by `max_source_chars`) + BM25 candidates + `purpose.md` scope + status of each candidate page. Outputs per-page action: `create`, `update`, `flag`, `skip`. **RULE 1b:** a page with `status='active'` is human-reviewed and authoritative — when the source provides a conflicting value, date, formula, or conclusion, the action must be `flag`, not `update`. The only exception is a source that adds a topic the page does not yet mention at all. Decision cache version: `v2`. |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. If no citation markers are present in the returned body, a `citation_pass4_no_markers` WARNING audit event is written — the page is still saved but the condition is flagged for re-ingest with a more capable model. Zero-citation results are not cached so a subsequent re-ingest can succeed. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic. **Suppressed when TF fallback was used** (a small corpus with broad coverage may legitimately return fewer than 3 candidates — that is expected, not a gap).\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n| Citation presence (Check 5b) | Page body has ≥ 50 words and zero `^[filename:L-L]` citation markers → WARNING. Diagnostic only — does not block promotion to `active`. Indicates the annotation pass failed to produce markers, usually a model-compatibility issue (use Gemini 2.5 Flash or higher for reliable citation annotation). |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n### ActionAgent\n\nDispatches action-intent queries from the chat UI (e.g. \"activate a draft page\", \"show lint report\", \"archive a stale page\", \"run scaffold\"). Parses the user's free-text intent into a structured action via an LLM extraction prompt, then executes the action against the wiki and returns a human-readable result. When the intent is ambiguous (e.g. \"activate a page\" without specifying which), sets `needs_clarification=True` and returns a prompt and candidate list for the web UI to render as chip buttons.\n\n### ScaffoldAgent\n\nGenerates and updates the wiki's structural pages. Reads all current wiki pages and asks the LLM to organise them into 5–8 domain-appropriate categories, then writes `wiki/index.md` (category index with wikilinks), `wiki/purpose.md` (scope statement), and `AGENTS.md` (ingest and query guidelines). Also stamps a `categories:` field on each page's YAML frontmatter. When `ROUTING.md` already exists, regenerates it to stay in sync with the updated index structure.\n\n### RewriteAgent\n\nRewrites follow-up questions in multi-turn conversations into self-contained, standalone form before BM25 retrieval. Converts context-dependent phrases (\"What came after that?\", \"Tell me more about his early life\") into explicit questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the correct pages without relying on conversation context. Only invoked when conversation history is non-empty. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### SummarizeAgent\n\nCompresses the oldest conversation turns into a single `[Session summary: …]` assistant turn when a session exceeds `conversation_history_turns`. Prevents unbounded context growth across long sessions. Emits a `notice` SSE event the first time compression occurs so the user can see that earlier context was condensed. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### ContextAgent\n\nBuilds token-budgeted context packs for MCP tool consumers. See [Section 20 — Context Packs](#20-context-packs) for full detail.\n\n### SearchDecomposeAgent\n\nDecomposes a web search intent into 1–4 terse keyword search strings optimised for authoritative source retrieval. Uses a separate prompt from `QueryAgent`'s question decomposition — search decomposition asks \"what distinct search strings would find the best sources?\" (terse keyword phrases) rather than \"what distinct questions does this ask?\" (natural-language sub-questions). See [Web search fan-out](#web-search-fan-out) in the IngestAgent section.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n| `session` | `.jsonl` | `claude session`, `codex session`, `cursor session`, `ai session`, `session history` | Extracts human-readable turns from AI coding session transcripts. Supports Claude Code JSONL format and Codex/Cursor format. Filters tool calls, thinking blocks, and sub-agent scaffolding. Short turns (< 20 assistant words, < 3 user words) are skipped. No external dependencies. |\n\n### Session Skill — AI Session History Ingestion\n\nThe `session` skill turns AI coding session history files (`.jsonl`) into searchable wiki pages capturing the problem-solving knowledge exchanged during the session.\n\n**Supported formats**\n\n| Format | File origin | Detection |\n|--------|-------------|-----------|\n| Claude Code | `~/.claude/projects//.jsonl` | `obj.type` ∈ `{\"user\", \"assistant\"}` with nested `obj.message` |\n| Codex / Cursor | Export from OpenAI Codex or Cursor IDE | `obj.role` at top level, no `message` wrapper |\n\nFormat is auto-detected from the first 30 parseable lines. Files that match neither format are parsed as Codex (fallback).\n\n**Filtering rules**\n\n| Content | Kept? | Reason |\n|---------|-------|--------|\n| User text ≥ 3 words | ✓ | Substantive question or instruction |\n| Assistant text ≥ 20 words | ✓ | Substantive answer |\n| User text < 3 words (`\"ok\"`, `\"yes\"`) | ✗ | Too terse to be useful |\n| Assistant text < 20 words | ✗ | Acknowledgement or one-liner |\n| `isSidechain: true` lines | ✗ | Sub-agent scaffolding (not the user's conversation) |\n| `tool_use` blocks | ✗ | Shell commands / file writes — not final output |\n| `tool_result` blocks | ✗ | Raw output — avoids leaking file contents or credentials |\n| `thinking` blocks | ✗ | Internal reasoning — not the final answer |\n| `permission-mode`, `system`, `last-prompt` lines | ✗ | Session metadata, not conversation |\n\nAfter extraction the text passes through Synthadoc's standard pre-LLM source sanitizer\n(zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars,\ninstruction-override phrases) — the same step applied to every PDF, DOCX, URL, and other source type.\nSee [§29 Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer).\n\n**Output format**\n\nEach kept turn is labelled and separated by `---`:\n\n```\n[USER]\nHow do I implement a sliding window algorithm in Python?\n\n---\n\n[ASSISTANT]\nA sliding window algorithm maintains a contiguous subarray by advancing two\npointers simultaneously…\n```\n\n**`suggested_slug`**\n\nThe skill sets `metadata[\"suggested_slug\"]` to `session-YYYY-MM-DD-` where the date comes from the file's `mtime` and the topic from the first substantive user turn (first 6 words, slugified). Example: `session-2026-07-15-how-do-i-implement-sliding`.\n\n**Limitations**\n\n- No chunking in v1.1 — very long sessions are truncated at `max_source_chars` (default 400 000 chars). Split large archives into individual `.jsonl` files.\n- Tool output is excluded by design — re-ingest the original source files if you need the file contents in the wiki.\n- Re-ingesting the same `.jsonl` file uses Synthadoc's standard source-hash dedup — no duplicate pages are created.\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n**Compound identifier expansion:** When a token contains underscores, the tokenizer emits both the compound form and each component as separate tokens (`capex_growth` → `[\"capex_growth\", \"capex\", \"growth\"]`). This expansion is applied at both index time and query time so that human-friendly queries match programmatic identifiers and vice versa. Leading/trailing underscores are stripped before splitting; no empty strings are emitted.\n\n**TF fallback for small corpora:** When all BM25 scores in a result set are zero or negative (IDF collapse on a corpus of ≤ 2 documents, or a term present in every page), the search system falls back to normalised term frequency (TF) scoring: `TF(doc, query) = Σcount(term, doc) / len(doc_tokens)`. Results are returned in the same format as BM25 results with a `tf_fallback: True` flag on each `SearchResult`. This field is recorded in the audit trail and consumed by gap detection (see below).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp, cascade_links_removed_from: [str]}` _(cascade field added v1.0.2)_ |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n| `GET` | `/graph` _(v1.0.0)_ | — | `{status, node_count, edge_count, cluster_count, nodes: [...], edges: [...]}` or `{status: \"computing\"}` on first call |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n**External file paths** (outside the wiki root) are supported when the request comes from\n`127.0.0.1` or `::1` and the payload includes `\"allow_external_paths\": true`. The CLI sets\nthis flag automatically for all local file sources. Remote clients cannot set this flag\n(the server ignores it for non-localhost connections) to prevent arbitrary file reads\nin server-exposed deployments.\n\nThis is required for ingesting files that live outside the wiki directory by design —\nfor example, Claude Code session transcripts at `~/.claude/projects//.jsonl`.\n\n**Client behaviour summary:**\n\n| Client | Sends `allow_external_paths` | Can ingest outside wiki root? |\n|--------|------------------------------|-------------------------------|\n| CLI (`synthadoc ingest `) | `true` (automatic, local file paths only) | Yes — server is localhost |\n| Obsidian plugin | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Web UI | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Direct HTTP API (localhost) | Set manually in request body | Yes — if request from 127.0.0.1/::1 |\n| Direct HTTP API (remote) | Ignored by server | No — server silently treats as `false` |\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 1.0.2\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n**Reading View default:** `synthadoc plugin install` and `synthadoc plugin upgrade` write `\"defaultViewMode\": \"preview\"` to `.obsidian/app.json` after copying the plugin files. This causes Obsidian to open new notes in Reading View by default, which is required for `^[filename:L-L]` citation chips to render. Only `defaultViewMode` is written; all other keys in `app.json` are preserved. If `app.json` is malformed JSON the write is skipped and a warning is logged — the install still succeeds.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports two things: dangling slugs (pages listed in ROUTING.md that no longer exist in the wiki) and unassigned slugs (pages that exist in index.md but are not listed in any ROUTING.md branch). Enabled only when ROUTING.md exists. **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n| `Graph: show knowledge graph` | Open the knowledge graph panel — Canvas force graph with type filter, hover tooltip, and click-to-open |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [-w wiki] — generate ROUTING.md from index.md branch structure\n│ ├── validate [-w wiki] — report dangling slugs and cross-branch duplicates\n│ └── clean [-w wiki] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [-w wiki]\n├── candidates\n│ ├── list [-w wiki]\n│ ├── promote |--all [-w wiki]\n│ └── discard |--all [-w wiki]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n# Citation pass (Pass 4) tuning — these two settings work together:\n# citation_source_lines — how many lines of the source the LLM sees when placing ^[...] markers.\n# Increase if lint reports out_of_range on long sources (transcripts, PDFs).\n# citation_max_tokens — output token budget for the annotated section returned by the LLM.\n# Increase if you raise citation_source_lines and have long wiki sections.\n# citation_source_lines = 400\n# citation_max_tokens = 8192\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Emit `logger.warning` in server log when per-job ingest cost exceeds this threshold; job continues normally |\n| `cost.hard_gate_usd` | float | `2.00` | Permanently fail the ingest job (DEAD) and record a `cost_gate_exceeded` audit event when per-job cost exceeds this threshold |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × (backup_count + 1)` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n| `ingest.max_source_chars` | int | `32000` | Character limit applied to each source before the LLM call. Sources exceeding this limit are truncated; the page's `sources:` frontmatter entry gets `truncated: true` and lint emits a warning. Override per-run with `--max-source-chars N`. _(v1.0.0)_ |\n| `ingest.citation_source_lines` | int | `400` | Number of source lines the LLM sees during Pass 4 (citation annotation). Increase if lint reports `out_of_range` on long sources such as transcripts or PDFs. Raise `citation_max_tokens` in proportion when increasing this value. |\n| `ingest.citation_max_tokens` | int | `8192` | Output token budget for the annotated section returned by Pass 4. Increase when `citation_source_lines` is raised or wiki sections are long, to avoid truncated annotations. |\n| `query.context_wiki_pct` | float | `0.60` | Fraction of the model context window reserved for wiki source pages. _(v1.0.0)_ |\n| `query.context_history_pct` | float | `0.20` | Fraction reserved for conversation history. _(v1.0.0)_ |\n| `query.context_system_pct` | float | `0.15` | Fraction reserved for system prompt and instructions. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_index_pct` | float | `0.05` | Fraction reserved for the wiki index preamble. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_window` | int | _(auto)_ | Override the built-in context window lookup for the configured model. Use when running a local or custom model whose window size is not in the built-in table. _(v1.0.0)_ |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement is wired into the ingest pipeline: `Orchestrator._guard_ingest_cost()` is called after each ingest LLM call with the returned cost, before the job is marked complete.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | `logger.warning` in server log; job continues normally |\n| `hard_gate_usd` | $2.00 | Server/ingest path: records `cost_gate_exceeded` audit event + permanently fails job (DEAD). Interactive CLI: prompts `Proceed? [y/N]` |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` by default. The bind address is configurable via `server.host` in `config.toml` (e.g. `host = \"0.0.0.0\"` to expose on all interfaces for LAN access). No built-in authentication — restrict via firewall when exposing beyond loopback. Remote access without a reverse proxy is not recommended on shared or networked machines.\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs and unassigned slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `-w ` / `--wiki ` to target a specific wiki.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; when all local source files are missing from disk the page is transitioned to `archived` (if only some sources are missing it is transitioned to `stale` instead). Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: lint evaluates all local (non-URL) source paths listed in the page's frontmatter. If **all** of them are missing from disk, the page is transitioned to `archived`. If only **some** are missing — meaning valid content from at least one source still exists — the page is transitioned to `stale` instead, preserving it for re-ingest rather than discarding it. _(v1.0.2: previously any single missing source triggered an immediate archive.)_\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Cascade link cleanup\n\nWhenever a page transitions to `archived` — whether triggered manually (`synthadoc lifecycle archive`), via the Obsidian Lifecycle modal, the MCP `synthadoc_lifecycle` tool, or automatically by lint detecting a missing source — Synthadoc immediately scans all other active pages and removes every `[[archived-slug]]` wikilink pointing to the now-archived page.\n\n- **Inline links** — `[[archived-slug]]` becomes plain text (the display text if an alias was used, otherwise the slug itself).\n- **List-item links** — a list item whose only content is `[[archived-slug]]` is dropped entirely.\n- **Archived pages are skipped** — pages already archived are not rewritten.\n- **System pages are skipped** — index, overview, dashboard, log, and purpose pages are excluded.\n\nThe cleanup runs synchronously before the response is returned, so the wiki is consistent immediately. The CLI and MCP responses include a `cascade_links_removed_from` field listing the affected slugs. During `lint run`, cascade is batched once after all auto-archive transitions complete, and the count is added to `dangling_links_removed` in the lint report.\n\nLint's existing periodic dangling-link cleanup (`lint run --scope orphans`) remains in place as a safety net for any dead links that predate this feature or arrive through other paths.\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 5) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 5 # turns to include in each request (default: 5; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp, cascade_links_removed_from: [str]}` or `{error, cascade_links_removed_from: []}` _(cascade field added v1.0.2)_ | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── hooks/ ← user hook scripts (always included)\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ ├── extracted/ ← text sidecars + PDF pagemaps (always included)\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `logs/`.\n\n> **v0.9.3 fix:** `hooks/` and `.synthadoc/extracted/` were previously missing from backups. Omitting `extracted/` caused the Source Viewer and Provenance citation modal to show \"Could not read source file\" after restore.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## 29. Pre-LLM Source Sanitizer\n\nEvery source document passes through a sanitizer immediately after text extraction, before any LLM call. The sanitizer removes six categories of content that could manipulate the LLM or degrade compilation quality:\n\n| Category | Action | Warning logged? |\n|---|---|---|\n| Zero-width characters (U+200B, U+200C, U+200D, U+FEFF) | Removed silently | No |\n| Bidi override characters (U+202A–U+202E, U+2066–U+2069) | Removed | Yes |\n| HTML comments (``) | Removed silently | No |\n| Hidden CSS spans (`display:none`, `visibility:hidden`) | Removed silently | No |\n| Base64 blobs ≥ 200 consecutive characters | Replaced with `[base64 content removed]` | Yes |\n| Instruction-override phrases (8 patterns: \"ignore previous instructions\", \"disregard the above\", \"override your system prompt\", etc.) | Replaced with `[redacted]` | Always |\n\nWhen content is removed and a warning is appropriate, the server logs a `WARN` entry:\n```\n[WARN] sanitizer stripped content from 'papers/survey.pdf': bidi overrides, instruction-override phrase\n```\n\nThis is a pure pre-processing step with zero LLM cost. Truncation (if configured) is applied after sanitization, so the character budget is consumed by clean text only.\n\n---\n\n## 30. Per-Source Truncation Flag\n\nLarge source documents are truncated before they reach the LLM. By default the limit is **32,000 characters** (~8,000 tokens), raised from the prior hardcoded 8,000-character limit. When a source exceeds this limit, its compiled wiki page records a `truncated: true` flag in its `sources:` frontmatter entry.\n\n### Configuration\n\n```toml\n[ingest]\nmax_source_chars = 32000 # default; raise for large PDFs or books\n```\n\n### Per-ingest override\n\nThe limit can be raised for a single ingest run without modifying the config:\n\n```bash\nsynthadoc ingest papers/large-textbook.pdf --max-source-chars 128000\n```\n\nThe same `max_source_chars` field is accepted by the HTTP `POST /jobs/ingest` body and the MCP `synthadoc_ingest` tool.\n\n### Lint surfacing\n\n`synthadoc lint` emits a warning for any page that has a truncated source:\n\n```\n[WARN] quantum-computing.md: source 'papers/large-survey.pdf' was truncated at ingest\n (source exceeded max_source_chars=32000 — 87,412 chars in source).\n To re-ingest with a higher limit (this source only):\n synthadoc ingest papers/large-survey.pdf --max-source-chars 128000\n To raise the limit for all future ingests:\n set [ingest] max_source_chars = 128000 in your config\n```\n\n---\n\n## 31. Proportional Context Budget\n\nQuery context is allocated proportionally to the configured model's context window, replacing the prior hard cap. The default allocation:\n\n| Slice | Default | Purpose |\n|---|---|---|\n| `context_wiki_pct` | 60% | Wiki page content (ranked by BM25 + vector score) |\n| `context_history_pct` | 20% | Chat turn history (newest-first, oldest dropped when over budget) |\n| `context_system_pct` | 15% | System prompt and purpose document |\n| `context_index_pct` | 5% | `ROUTING.md` and search index |\n\nPercentages must sum to ≤ 100. Configure in `[query]`:\n\n```toml\n[query]\ncontext_window = 0 # 0 = auto-detect from known model map\ncontext_wiki_pct = 60\ncontext_history_pct = 20\ncontext_system_pct = 15\ncontext_index_pct = 5\n```\n\n> **Note:** `context_system_pct` and `context_index_pct` are parsed and validated but not yet enforced as hard caps in this release — the system prompt uses a conservative fixed limit, which is well within the configured 15% slice for all supported models. Full per-slice enforcement is planned for v1.1.\n\n### Model context window map\n\nSynthadoc ships a built-in prefix-matched table:\n\n| Model | Context window |\n|---|---|\n| `claude-opus-4*`, `claude-sonnet-4*`, `claude-haiku-4*` | 200,000 tokens |\n| `gpt-4o*`, `gpt-4-turbo*` | 128,000 tokens |\n| `gpt-4` (exact) | 8,192 tokens |\n| `gpt-3.5-turbo*` | 16,385 tokens |\n| Unknown / fallback | 128,000 tokens |\n\nSet `context_window = N` in config to override the map (useful for local Ollama models with non-standard context sizes).\n\nAt the defaults on a 200k-token model: wiki slice ≈ 480,000 chars — far more than any practical query needs. On constrained models, the budget scales down automatically so the same config works across model families.\n\n---\n\n## 32. Knowledge Graph\n\nSynthadoc computes a wikilink graph across all active and draft pages during every lint run. The graph is stored in two tables in `audit.db`:\n\n```sql\nCREATE TABLE graph_nodes (\n slug TEXT PRIMARY KEY,\n cluster_id INTEGER NOT NULL,\n updated_at TEXT NOT NULL\n);\n\nCREATE TABLE graph_edges (\n from_slug TEXT NOT NULL,\n to_slug TEXT NOT NULL,\n weight INTEGER NOT NULL DEFAULT 1,\n edge_type TEXT NOT NULL DEFAULT 'mixed',\n PRIMARY KEY (from_slug, to_slug)\n);\n```\n\nEdge weight is a composite signal from two sources:\n\n| Edge type | Source | Weight contribution |\n|-----------|--------|---------------------|\n| `wikilink` | `[[slug]]` occurrences in page body | +1 per occurrence |\n| `co_source` | pages compiled from the same source file (matched by hash) | +2 per shared source |\n\nMost edges are `mixed` (have both types). Pure `co_source` edges surface hidden relationships — pages compiled from the same source document are linked immediately after ingest, before any wikilinks are created. `edge_type` is stored on each edge so the web UI can render co-source edges with a different visual style.\n\nCluster IDs are assigned by Louvain community detection — pages that link densely to each other end up in the same cluster.\n\n### REST API\n\n**`GET /graph`** — returns the full graph with per-node enrichment:\n\n```json\n{\n \"status\": \"ready\",\n \"node_count\": 45,\n \"edge_count\": 123,\n \"cluster_count\": 4,\n \"nodes\": [\n {\"slug\": \"quantum-error-correction\", \"title\": \"Quantum Error Correction\",\n \"type\": \"concept\", \"state\": \"active\", \"cluster_id\": 2}\n ],\n \"edges\": [\n {\"from\": \"quantum-error-correction\", \"to\": \"shor-algorithm\", \"weight\": 3, \"edge_type\": \"mixed\"}\n ]\n}\n```\n\nWhen the graph tables are empty (first call after upgrade, or before the first lint run), the server returns `{\"status\": \"computing\"}` and starts a background build immediately. The web UI polls every 2 seconds and renders the graph when the status becomes `\"ready\"`.\n\n### Web UI\n\nThe **Graph** tab (alongside Chat in the top nav) renders a D3.js force-directed graph. Nodes are colored by cluster. Edge thickness is proportional to weight (1–4 px). Pure `co_source` edges (no wikilink) are rendered with a dashed stroke, making hidden source-based relationships visually distinct from explicit cross-links. Clicking a node opens a sidebar showing the page title, type and lifecycle state badges, cluster assignment, and an **\"Ask about this →\"** button that switches to the Chat tab with a pre-filled query.\n\nControls: zoom in/out (scroll or pinch), drag nodes, filter by page type.\n\n---\n\n## Customization\n\nSynthadoc exposes four extension points — all hot-loaded, no server restart required.\n\n### Custom skills (new file formats)\n\nSubclass `BaseSkill` (Apache-2.0 — no AGPL obligation on your skill code), drop the folder in `/skills/` or `~/.synthadoc/skills/`, and Synthadoc picks it up on the next ingest. Skills match by file extension or intent prefix and support any Unicode text, including CJK prefixes.\n\n→ Full interface, manifest format, and examples: [§5 Skills System](#5-skills-system) and [§17 Plugin Development Guide](#17-plugin-development-guide)\n\n### Custom LLM providers\n\nSubclass `LLMProvider` from `synthadoc/providers/base.py` (Apache-2.0) and place the file in `~/.synthadoc/providers/` or the wiki `providers/` directory. Switch active provider with one config line.\n\n→ Full interface and wiring instructions: [§10 Configuration — Provider switching](#10-configuration)\n\n### Hooks\n\nShell scripts (any language) that fire on `on_ingest_complete` and `on_lint_complete`. Receive a JSON context on stdin. Set `blocking = true` to gate the operation on the hook's exit code — useful for CI pipelines and quality gates.\n\n→ Full event schema, context JSON, and blocking behaviour: [§11 Hook System](#11-hook-system)\n\n### Cache control\n\nThree cache layers (embedding, LLM response, provider prompt cache) invalidate automatically on source-file change (SHA-256). Force a fresh call with `--force`, or wipe all cached responses with `synthadoc cache clear -w `.\n\n→ Layer-by-layer breakdown and invalidation rules: [§12 Cache System](#12-cache-system)\n\n### Multi-Platform Agent Skill Files\n\nEvery Synthadoc wiki ships three companion files that give AI coding agents a complete, self-contained operating guide for the wiki:\n\n| File | Platform |\n|---|---|\n| `AGENTS.md` | Codex CLI, OpenCode, and any agent that reads the AGENTS.md convention |\n| `CLAUDE.md` | Claude Code (this session's tool) — highest-priority instruction source in Claude's hierarchy |\n| `GEMINI.md` | Gemini CLI — relevant because Synthadoc's default LLM provider is `gemini-2.5-flash-lite` |\n\nAll three files share identical body content generated from the same template; they differ only in their H1 heading so each agent recognises its own file first. The body covers:\n\n- **Domain guidelines** — LLM-generated bullet list of domain-specific ingest and query rules (produced by `scaffold`)\n- **Quick reference table** — every common CLI command with the correct `synthadoc` syntax\n- **Server startup** — how to start the server and verify it is running\n- **Ingest examples** — local file, URL, YouTube, agent session `.jsonl`, force reingest, dry-run\n- **Query** — basic and streaming query syntax; citation marker format\n- **Lint** — when to run and what each check does\n- **Lifecycle** — transition commands with state names\n- **Page schema** — frontmatter structure with all fields\n- **MCP tools table** — all 12 tools with purpose descriptions\n\n**How the files are created and updated:**\n\n1. `synthadoc init` — writes all three files with default domain guidelines and the wiki's configured port.\n2. `synthadoc scaffold` — regenerates all three with LLM-produced domain guidelines derived from the current wiki state. The `scaffold` command prints a confirmation line for each file.\n\n**Use cases:**\n\n- Open Claude Code in a Synthadoc wiki directory and it immediately knows how to ingest, query, lint, and manage lifecycle — no additional setup. Claude Code can also start the server in background mode (`synthadoc serve -w -b`), stop it cleanly, and orchestrate multi-step workflows, effectively acting as a control plane for the wiki.\n- Gemini CLI reads `GEMINI.md` and can run `synthadoc` commands with the correct flags without asking the user for help.\n- CI pipelines with Codex CLI use `AGENTS.md` to drive automated lint and lifecycle transitions on PR merge.\n- Agent session files produced by Claude Code (`.jsonl` under `~/.claude/projects/`) can themselves be ingested into the wiki, capturing the reasoning behind design decisions alongside the source documents.\n\n→ See [README.md — Interfaces & Integration](../README.md#interfaces--integration) for the feature comparison row and quick-start instructions.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v1.1.0 (in progress)\n\n- **Weighted knowledge graph edges** — graph edges now carry two signals: wikilink occurrences (+1 each) and co-source connections — pages compiled from the same source file (matched by SHA-256 hash) — (+2 per shared source). `edge_type` field added to `graph_edges` table (`wikilink`, `co_source`, or `mixed`). `GET /graph` exposes `edge_type` per edge. Web UI renders edge thickness proportional to weight (1–4 px) and dashed lines for pure co-source edges. Co-source edges appear immediately after ingest — before any wikilinks — surfacing hidden relationships in the graph. Schema migrated to version 3.\n- **Multi-platform agent skill files** — `synthadoc init` now writes `CLAUDE.md`, `AGENTS.md`, and `GEMINI.md` at wiki root; all three carry the same complete CLI reference so Claude Code, Codex CLI, OpenCode, Pi, and Gemini CLI users get first-class guidance without manual setup.\n- **Session history ingestion skill** — `.jsonl` session files from Claude Code and Codex CLI are ingested as wiki pages; human turns and substantive assistant responses extracted; tool calls, thinking blocks, and sub-agent scaffolding skipped; format auto-detected; `suggested_slug` derived from session date and first user message.\n\n### v1.0.2\n\n- **Cascade link cleanup on archive** — archiving any page immediately removes every `[[slug]]` wikilink pointing to it from all other active pages, without waiting for the next `lint run`. Inline links become plain text; list-item-only links are dropped entirely. The HTTP `POST /lifecycle/transition` and MCP `synthadoc_lifecycle` responses now include a `cascade_links_removed_from` field listing the rewritten page slugs. During `lint run`, cascade runs once after all auto-archive transitions complete and the count is included in the lint report's `dangling_links_removed` field. System pages (index, dashboard, purpose, log, overview) and already-archived pages are skipped. The existing dangling-link lint pass remains as a safety net for dead links that predate this release.\n- **Multi-source partial-archive fix** — when a page has multiple local source files and only some are missing from disk, lint now transitions the page to `stale` (not `archived`), preserving it for re-ingest. The page is only archived when every local source file is gone. URL sources are not affected by this check.\n\n### v1.0.1\n\n- **Scaffold JSON extraction hardened** — replaced the greedy regex used to extract the LLM's JSON response with a brace-balanced parser; eliminates false-positive matches on `[[wikilink]]` syntax that previously caused `json.loads()` failures and silent scaffold corruption\n- **Generated file validation** — scaffold now validates every LLM-generated file (index, purpose, AGENTS.md) for minimum length, required frontmatter, and `[[wikilink]]` presence before writing to disk; validation failure marks the job as failed with a structured error message rather than silently writing a malformed page\n- **ROUTING.md regeneration validation** — `routing init` and scheduled routing regeneration now validate the regenerated file for minimum line count and anchor presence before applying it; prevents a corrupt or truncated ROUTING.md from entering the wiki\n- **Scaffold failure reporting** — scaffold skip and failure reasons are now surfaced in the terminal output and job error field; previously silent failures (e.g. LLM timeout, JSON parse error) are now visible without tailing the JSON log\n- **Install-time scaffold removed** — `synthadoc install` no longer runs the LLM scaffold call during installation; scaffold is a separate step (`synthadoc scaffold`) that the user runs after setting their API key, avoiding confusing API-key-not-set errors at install time\n- **AquaFlow LLM evaluation** — 15-question M&A due-diligence benchmark across five models (MiniMax-Think M3, Claude Opus 4.8, Claude Sonnet 4.6, DeepSeek-R1, Qwen Plus); full report at `docs/example/aquaflow/evaluation/report/llm-query-benchmark.md`\n\n### v1.0.0\n\n- **Pre-LLM source sanitizer** — every source document passes through a sanitizer before any LLM call; removes six categories of potentially harmful content (zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars, instruction-override phrases); a `WARN` log entry is emitted when bidi overrides or instruction-override phrases are stripped; zero LLM cost; applied before truncation so the character budget is consumed by clean text only\n- **Per-source truncation flag** — default character limit raised from 8,000 to 32,000 characters per source (~8,000 tokens); truncated sources are flagged with `truncated: true` in the page's `sources:` frontmatter; `--max-source-chars N` CLI flag and matching `POST /jobs/ingest` body field and MCP `synthadoc_ingest` parameter override the limit per run; `synthadoc lint` emits a warning for any page with a truncated source and suggests the override command; configurable via `[ingest] max_source_chars` in `config.toml`\n- **Proportional context budget** — query context is allocated proportionally to the configured model's context window; four configurable slices: `context_wiki_pct` (60%), `context_history_pct` (20%), `context_system_pct` (15%), `context_index_pct` (5%); built-in prefix-matched model context window table covers Claude 4, GPT-4o/turbo/3.5, and GPT-4; unknown models fall back to 128,000 tokens; `context_window = N` in `[query]` overrides the table for local or custom models\n- **Knowledge graph** — wikilink graph computed across all active and draft pages during every lint run; stored in `graph_nodes` and `graph_edges` tables in `audit.db`; edge weight = number of `[[wikilink]]` references between two pages; cluster IDs assigned by Louvain community detection; `GET /graph` REST endpoint returns nodes with per-node enrichment (title, type, state, cluster) and edges; first call after upgrade triggers a background build and returns `{\"status\":\"computing\"}`; web UI **Graph** tab (D3.js force-directed graph) — nodes colored by cluster, click to see page details and an \"Ask about this →\" button that opens a pre-filled chat query\n- **Citation accuracy & rendering improvements** — six correctness fixes to the ingest citation pipeline: (1) full source text passed to the decision LLM so citations can reference any line in the document; (2) `_annotate_citations` bugs A–F resolved (false-positive matches, off-by-one line ranges, duplicate markers, escaped bracket handling, multi-paragraph span errors, and source filename normalisation); (3) `_has_citations` check added — lint warns and suggests a model upgrade when a page has zero citation markers after synthesis; (4) `Key Data` section extraction hardened so numerical facts, rates, and formulas are preserved verbatim across the annotation pass; (5) `obsidianCitationsToGfm` rendering fixed in the web UI — citation superscripts now display correctly in the chat panel alongside streamed answers; (6) reading view set as the default Obsidian display mode for wiki pages\n\n### v0.9.3\n\n- **Backup now includes `hooks/` and `.synthadoc/extracted/`** — both directories were silently omitted from backups. `hooks/` contains user-customised hook scripts; `.synthadoc/extracted/` contains per-source text sidecars and PDF pagemaps written by the ingest agent. Without these, the Source Viewer and Provenance citation modal showed \"Could not read source file\" after restore.\n- **PyPI publish workflow** — `publish.yml` triggers on GitHub Release, builds the wheel with hatchling, and uploads to PyPI via OIDC trusted publishing (no API token stored). `pip install synthadoc` now works.\n- **Web UI bundled in pip package** — `synthadoc/data/web-ui/dist/` is now included in the wheel. `synthadoc web` previously returned 503 on pip-installed instances because the React dist was not packaged.\n- **CI sync checks** — PRs now fail if `synthadoc/data/obsidian-plugin/`, `synthadoc/data/web-ui/`, or `README-pypi.md` are out of sync with their source. All sync-check steps use `shell: bash` for Windows runner compatibility.\n- **`bump_version.py` extended** — now also updates `synthadoc/data/obsidian-plugin/manifest.json` alongside the source manifest, preventing version drift between the bundled copy and the plugin source.\n\n### v0.9.2\n\n- **`synthadoc/data/web-ui/dist/` packaged** — web UI dist files committed to the repo and bundled in the wheel via hatchling artifacts.\n- **`scripts/sync_web_ui.py`** — helper script to sync `web-ui/dist/` into `synthadoc/data/web-ui/dist/` after `npm run build`.\n\n### v0.9.1\n\n- **PyPI metadata** — added `readme`, `keywords`, `classifiers`, and `[project.urls]` to `pyproject.toml` for correct PyPI rendering.\n- **`README-pypi.md`** — generated from `README.md` by `scripts/prepare_pypi_readme.py`; strips `pypi-strip` blocks and rewrites relative links to absolute GitHub URLs.\n- **Branch protection compatibility** — removed post-merge `build-plugin` CI job that pushed compiled plugin directly to `main`. Plugin sync is now enforced in PRs: CI fails if `synthadoc/data/obsidian-plugin/` is out of sync.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 5). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n"}, "9c7e222b73c6bc68fe0cf5a79096e50c3afc83f7": {"t": 1785767257.0020487, "n": 206487, "h": "762cd3054e140598", "c": "# Synthadoc — Design Document\n\n**Version:** 1.0.2 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n29. [Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer)\n30. [Per-Source Truncation Flag](#30-per-source-truncation-flag)\n31. [Proportional Context Budget](#31-proportional-context-budget)\n32. [Knowledge Graph](#32-knowledge-graph)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the full source text (bounded by `max_source_chars`) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `flag` / `skip`). Pages with `status='active'` are protected — sources that conflict with them trigger `flag` rather than `update` (RULE 1b).\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Orchestrator calls `_guard_ingest_cost()` on the returned cost: soft warn → `logger.warning` only; hard gate → `cost_gate_exceeded` audit event + `fail_permanent` (job → DEAD). On success, job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 0b — Key Data Extraction | None (regex) | Deterministic pre-processor: extracts numbers, percentages, currency amounts, ratios, formulas, underscore identifiers, and date ranges from source text via regex; appends them as a `[Key Data — extracted by pre-processor]` section before any LLM sees the text. Zero token cost; cannot hallucinate. Anchors the synthesis LLM to all quantitative candidates. |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads the full source text (bounded by `max_source_chars`) + BM25 candidates + `purpose.md` scope + status of each candidate page. Outputs per-page action: `create`, `update`, `flag`, `skip`. **RULE 1b:** a page with `status='active'` is human-reviewed and authoritative — when the source provides a conflicting value, date, formula, or conclusion, the action must be `flag`, not `update`. The only exception is a source that adds a topic the page does not yet mention at all. Decision cache version: `v2`. |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. If no citation markers are present in the returned body, a `citation_pass4_no_markers` WARNING audit event is written — the page is still saved but the condition is flagged for re-ingest with a more capable model. Zero-citation results are not cached so a subsequent re-ingest can succeed. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic. **Suppressed when TF fallback was used** (a small corpus with broad coverage may legitimately return fewer than 3 candidates — that is expected, not a gap).\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n| Citation presence (Check 5b) | Page body has ≥ 50 words and zero `^[filename:L-L]` citation markers → WARNING. Diagnostic only — does not block promotion to `active`. Indicates the annotation pass failed to produce markers, usually a model-compatibility issue (use Gemini 2.5 Flash or higher for reliable citation annotation). |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n### ActionAgent\n\nDispatches action-intent queries from the chat UI (e.g. \"activate a draft page\", \"show lint report\", \"archive a stale page\", \"run scaffold\"). Parses the user's free-text intent into a structured action via an LLM extraction prompt, then executes the action against the wiki and returns a human-readable result. When the intent is ambiguous (e.g. \"activate a page\" without specifying which), sets `needs_clarification=True` and returns a prompt and candidate list for the web UI to render as chip buttons.\n\n### ScaffoldAgent\n\nGenerates and updates the wiki's structural pages. Reads all current wiki pages and asks the LLM to organise them into 5–8 domain-appropriate categories, then writes `wiki/index.md` (category index with wikilinks), `wiki/purpose.md` (scope statement), and `AGENTS.md` (ingest and query guidelines). Also stamps a `categories:` field on each page's YAML frontmatter. When `ROUTING.md` already exists, regenerates it to stay in sync with the updated index structure.\n\n### RewriteAgent\n\nRewrites follow-up questions in multi-turn conversations into self-contained, standalone form before BM25 retrieval. Converts context-dependent phrases (\"What came after that?\", \"Tell me more about his early life\") into explicit questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the correct pages without relying on conversation context. Only invoked when conversation history is non-empty. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### SummarizeAgent\n\nCompresses the oldest conversation turns into a single `[Session summary: …]` assistant turn when a session exceeds `conversation_history_turns`. Prevents unbounded context growth across long sessions. Emits a `notice` SSE event the first time compression occurs so the user can see that earlier context was condensed. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### ContextAgent\n\nBuilds token-budgeted context packs for MCP tool consumers. See [Section 20 — Context Packs](#20-context-packs) for full detail.\n\n### SearchDecomposeAgent\n\nDecomposes a web search intent into 1–4 terse keyword search strings optimised for authoritative source retrieval. Uses a separate prompt from `QueryAgent`'s question decomposition — search decomposition asks \"what distinct search strings would find the best sources?\" (terse keyword phrases) rather than \"what distinct questions does this ask?\" (natural-language sub-questions). See [Web search fan-out](#web-search-fan-out) in the IngestAgent section.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n| `session` | `.jsonl` | `claude session`, `codex session`, `cursor session`, `ai session`, `session history` | Extracts human-readable turns from AI coding session transcripts. Supports Claude Code JSONL format and Codex/Cursor format. Filters tool calls, thinking blocks, and sub-agent scaffolding. Short turns (< 20 assistant words, < 3 user words) are skipped. No external dependencies. |\n\n### Session Skill — AI Session History Ingestion\n\nThe `session` skill turns AI coding session history files (`.jsonl`) into searchable wiki pages capturing the problem-solving knowledge exchanged during the session.\n\n**Supported formats**\n\n| Format | File origin | Detection |\n|--------|-------------|-----------|\n| Claude Code | `~/.claude/projects//.jsonl` | `obj.type` ∈ `{\"user\", \"assistant\"}` with nested `obj.message` |\n| Codex / Cursor | Export from OpenAI Codex or Cursor IDE | `obj.role` at top level, no `message` wrapper |\n\nFormat is auto-detected from the first 30 parseable lines. Files that match neither format are parsed as Codex (fallback).\n\n**Filtering rules**\n\n| Content | Kept? | Reason |\n|---------|-------|--------|\n| User text ≥ 3 words | ✓ | Substantive question or instruction |\n| Assistant text ≥ 20 words | ✓ | Substantive answer |\n| User text < 3 words (`\"ok\"`, `\"yes\"`) | ✗ | Too terse to be useful |\n| Assistant text < 20 words | ✗ | Acknowledgement or one-liner |\n| `isSidechain: true` lines | ✗ | Sub-agent scaffolding (not the user's conversation) |\n| `tool_use` blocks | ✗ | Shell commands / file writes — not final output |\n| `tool_result` blocks | ✗ | Raw output — avoids leaking file contents or credentials |\n| `thinking` blocks | ✗ | Internal reasoning — not the final answer |\n| `permission-mode`, `system`, `last-prompt` lines | ✗ | Session metadata, not conversation |\n\nAfter extraction the text passes through Synthadoc's standard pre-LLM source sanitizer\n(zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars,\ninstruction-override phrases) — the same step applied to every PDF, DOCX, URL, and other source type.\nSee [§29 Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer).\n\n**Output format**\n\nEach kept turn is labelled and separated by `---`:\n\n```\n[USER]\nHow do I implement a sliding window algorithm in Python?\n\n---\n\n[ASSISTANT]\nA sliding window algorithm maintains a contiguous subarray by advancing two\npointers simultaneously…\n```\n\n**`suggested_slug`**\n\nThe skill sets `metadata[\"suggested_slug\"]` to `session-YYYY-MM-DD-` where the date comes from the file's `mtime` and the topic from the first substantive user turn (first 6 words, slugified). Example: `session-2026-07-15-how-do-i-implement-sliding`.\n\n**Limitations**\n\n- No chunking in v1.1 — very long sessions are truncated at `max_source_chars` (default 400 000 chars). Split large archives into individual `.jsonl` files.\n- Tool output is excluded by design — re-ingest the original source files if you need the file contents in the wiki.\n- Re-ingesting the same `.jsonl` file uses Synthadoc's standard source-hash dedup — no duplicate pages are created.\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n**Compound identifier expansion:** When a token contains underscores, the tokenizer emits both the compound form and each component as separate tokens (`capex_growth` → `[\"capex_growth\", \"capex\", \"growth\"]`). This expansion is applied at both index time and query time so that human-friendly queries match programmatic identifiers and vice versa. Leading/trailing underscores are stripped before splitting; no empty strings are emitted.\n\n**TF fallback for small corpora:** When all BM25 scores in a result set are zero or negative (IDF collapse on a corpus of ≤ 2 documents, or a term present in every page), the search system falls back to normalised term frequency (TF) scoring: `TF(doc, query) = Σcount(term, doc) / len(doc_tokens)`. Results are returned in the same format as BM25 results with a `tf_fallback: True` flag on each `SearchResult`. This field is recorded in the audit trail and consumed by gap detection (see below).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp, cascade_links_removed_from: [str]}` _(cascade field added v1.0.2)_ |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n| `GET` | `/graph` _(v1.0.0)_ | — | `{status, node_count, edge_count, cluster_count, nodes: [...], edges: [...]}` or `{status: \"computing\"}` on first call |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n**External file paths** (outside the wiki root) are supported when the request comes from\n`127.0.0.1` or `::1` and the payload includes `\"allow_external_paths\": true`. The CLI sets\nthis flag automatically for all local file sources. Remote clients cannot set this flag\n(the server ignores it for non-localhost connections) to prevent arbitrary file reads\nin server-exposed deployments.\n\nThis is required for ingesting files that live outside the wiki directory by design —\nfor example, Claude Code session transcripts at `~/.claude/projects//.jsonl`.\n\n**Client behaviour summary:**\n\n| Client | Sends `allow_external_paths` | Can ingest outside wiki root? |\n|--------|------------------------------|-------------------------------|\n| CLI (`synthadoc ingest `) | `true` (automatic, local file paths only) | Yes — server is localhost |\n| Obsidian plugin | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Web UI | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Direct HTTP API (localhost) | Set manually in request body | Yes — if request from 127.0.0.1/::1 |\n| Direct HTTP API (remote) | Ignored by server | No — server silently treats as `false` |\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 1.0.2\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n**Reading View default:** `synthadoc plugin install` and `synthadoc plugin upgrade` write `\"defaultViewMode\": \"preview\"` to `.obsidian/app.json` after copying the plugin files. This causes Obsidian to open new notes in Reading View by default, which is required for `^[filename:L-L]` citation chips to render. Only `defaultViewMode` is written; all other keys in `app.json` are preserved. If `app.json` is malformed JSON the write is skipped and a warning is logged — the install still succeeds.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports two things: dangling slugs (pages listed in ROUTING.md that no longer exist in the wiki) and unassigned slugs (pages that exist in index.md but are not listed in any ROUTING.md branch). Enabled only when ROUTING.md exists. **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [-w wiki] — generate ROUTING.md from index.md branch structure\n│ ├── validate [-w wiki] — report dangling slugs and cross-branch duplicates\n│ └── clean [-w wiki] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [-w wiki]\n├── candidates\n│ ├── list [-w wiki]\n│ ├── promote |--all [-w wiki]\n│ └── discard |--all [-w wiki]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n# Citation pass (Pass 4) tuning — these two settings work together:\n# citation_source_lines — how many lines of the source the LLM sees when placing ^[...] markers.\n# Increase if lint reports out_of_range on long sources (transcripts, PDFs).\n# citation_max_tokens — output token budget for the annotated section returned by the LLM.\n# Increase if you raise citation_source_lines and have long wiki sections.\n# citation_source_lines = 400\n# citation_max_tokens = 8192\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Emit `logger.warning` in server log when per-job ingest cost exceeds this threshold; job continues normally |\n| `cost.hard_gate_usd` | float | `2.00` | Permanently fail the ingest job (DEAD) and record a `cost_gate_exceeded` audit event when per-job cost exceeds this threshold |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × (backup_count + 1)` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n| `ingest.max_source_chars` | int | `32000` | Character limit applied to each source before the LLM call. Sources exceeding this limit are truncated; the page's `sources:` frontmatter entry gets `truncated: true` and lint emits a warning. Override per-run with `--max-source-chars N`. _(v1.0.0)_ |\n| `ingest.citation_source_lines` | int | `400` | Number of source lines the LLM sees during Pass 4 (citation annotation). Increase if lint reports `out_of_range` on long sources such as transcripts or PDFs. Raise `citation_max_tokens` in proportion when increasing this value. |\n| `ingest.citation_max_tokens` | int | `8192` | Output token budget for the annotated section returned by Pass 4. Increase when `citation_source_lines` is raised or wiki sections are long, to avoid truncated annotations. |\n| `query.context_wiki_pct` | float | `0.60` | Fraction of the model context window reserved for wiki source pages. _(v1.0.0)_ |\n| `query.context_history_pct` | float | `0.20` | Fraction reserved for conversation history. _(v1.0.0)_ |\n| `query.context_system_pct` | float | `0.15` | Fraction reserved for system prompt and instructions. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_index_pct` | float | `0.05` | Fraction reserved for the wiki index preamble. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_window` | int | _(auto)_ | Override the built-in context window lookup for the configured model. Use when running a local or custom model whose window size is not in the built-in table. _(v1.0.0)_ |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement is wired into the ingest pipeline: `Orchestrator._guard_ingest_cost()` is called after each ingest LLM call with the returned cost, before the job is marked complete.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | `logger.warning` in server log; job continues normally |\n| `hard_gate_usd` | $2.00 | Server/ingest path: records `cost_gate_exceeded` audit event + permanently fails job (DEAD). Interactive CLI: prompts `Proceed? [y/N]` |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` by default. The bind address is configurable via `server.host` in `config.toml` (e.g. `host = \"0.0.0.0\"` to expose on all interfaces for LAN access). No built-in authentication — restrict via firewall when exposing beyond loopback. Remote access without a reverse proxy is not recommended on shared or networked machines.\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs and unassigned slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `-w ` / `--wiki ` to target a specific wiki.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; when all local source files are missing from disk the page is transitioned to `archived` (if only some sources are missing it is transitioned to `stale` instead). Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: lint evaluates all local (non-URL) source paths listed in the page's frontmatter. If **all** of them are missing from disk, the page is transitioned to `archived`. If only **some** are missing — meaning valid content from at least one source still exists — the page is transitioned to `stale` instead, preserving it for re-ingest rather than discarding it. _(v1.0.2: previously any single missing source triggered an immediate archive.)_\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Cascade link cleanup\n\nWhenever a page transitions to `archived` — whether triggered manually (`synthadoc lifecycle archive`), via the Obsidian Lifecycle modal, the MCP `synthadoc_lifecycle` tool, or automatically by lint detecting a missing source — Synthadoc immediately scans all other active pages and removes every `[[archived-slug]]` wikilink pointing to the now-archived page.\n\n- **Inline links** — `[[archived-slug]]` becomes plain text (the display text if an alias was used, otherwise the slug itself).\n- **List-item links** — a list item whose only content is `[[archived-slug]]` is dropped entirely.\n- **Archived pages are skipped** — pages already archived are not rewritten.\n- **System pages are skipped** — index, overview, dashboard, log, and purpose pages are excluded.\n\nThe cleanup runs synchronously before the response is returned, so the wiki is consistent immediately. The CLI and MCP responses include a `cascade_links_removed_from` field listing the affected slugs. During `lint run`, cascade is batched once after all auto-archive transitions complete, and the count is added to `dangling_links_removed` in the lint report.\n\nLint's existing periodic dangling-link cleanup (`lint run --scope orphans`) remains in place as a safety net for any dead links that predate this feature or arrive through other paths.\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 5) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 5 # turns to include in each request (default: 5; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp, cascade_links_removed_from: [str]}` or `{error, cascade_links_removed_from: []}` _(cascade field added v1.0.2)_ | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── hooks/ ← user hook scripts (always included)\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ ├── extracted/ ← text sidecars + PDF pagemaps (always included)\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `logs/`.\n\n> **v0.9.3 fix:** `hooks/` and `.synthadoc/extracted/` were previously missing from backups. Omitting `extracted/` caused the Source Viewer and Provenance citation modal to show \"Could not read source file\" after restore.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## 29. Pre-LLM Source Sanitizer\n\nEvery source document passes through a sanitizer immediately after text extraction, before any LLM call. The sanitizer removes six categories of content that could manipulate the LLM or degrade compilation quality:\n\n| Category | Action | Warning logged? |\n|---|---|---|\n| Zero-width characters (U+200B, U+200C, U+200D, U+FEFF) | Removed silently | No |\n| Bidi override characters (U+202A–U+202E, U+2066–U+2069) | Removed | Yes |\n| HTML comments (``) | Removed silently | No |\n| Hidden CSS spans (`display:none`, `visibility:hidden`) | Removed silently | No |\n| Base64 blobs ≥ 200 consecutive characters | Replaced with `[base64 content removed]` | Yes |\n| Instruction-override phrases (8 patterns: \"ignore previous instructions\", \"disregard the above\", \"override your system prompt\", etc.) | Replaced with `[redacted]` | Always |\n\nWhen content is removed and a warning is appropriate, the server logs a `WARN` entry:\n```\n[WARN] sanitizer stripped content from 'papers/survey.pdf': bidi overrides, instruction-override phrase\n```\n\nThis is a pure pre-processing step with zero LLM cost. Truncation (if configured) is applied after sanitization, so the character budget is consumed by clean text only.\n\n---\n\n## 30. Per-Source Truncation Flag\n\nLarge source documents are truncated before they reach the LLM. By default the limit is **32,000 characters** (~8,000 tokens), raised from the prior hardcoded 8,000-character limit. When a source exceeds this limit, its compiled wiki page records a `truncated: true` flag in its `sources:` frontmatter entry.\n\n### Configuration\n\n```toml\n[ingest]\nmax_source_chars = 32000 # default; raise for large PDFs or books\n```\n\n### Per-ingest override\n\nThe limit can be raised for a single ingest run without modifying the config:\n\n```bash\nsynthadoc ingest papers/large-textbook.pdf --max-source-chars 128000\n```\n\nThe same `max_source_chars` field is accepted by the HTTP `POST /jobs/ingest` body and the MCP `synthadoc_ingest` tool.\n\n### Lint surfacing\n\n`synthadoc lint` emits a warning for any page that has a truncated source:\n\n```\n[WARN] quantum-computing.md: source 'papers/large-survey.pdf' was truncated at ingest\n (source exceeded max_source_chars=32000 — 87,412 chars in source).\n To re-ingest with a higher limit (this source only):\n synthadoc ingest papers/large-survey.pdf --max-source-chars 128000\n To raise the limit for all future ingests:\n set [ingest] max_source_chars = 128000 in your config\n```\n\n---\n\n## 31. Proportional Context Budget\n\nQuery context is allocated proportionally to the configured model's context window, replacing the prior hard cap. The default allocation:\n\n| Slice | Default | Purpose |\n|---|---|---|\n| `context_wiki_pct` | 60% | Wiki page content (ranked by BM25 + vector score) |\n| `context_history_pct` | 20% | Chat turn history (newest-first, oldest dropped when over budget) |\n| `context_system_pct` | 15% | System prompt and purpose document |\n| `context_index_pct` | 5% | `ROUTING.md` and search index |\n\nPercentages must sum to ≤ 100. Configure in `[query]`:\n\n```toml\n[query]\ncontext_window = 0 # 0 = auto-detect from known model map\ncontext_wiki_pct = 60\ncontext_history_pct = 20\ncontext_system_pct = 15\ncontext_index_pct = 5\n```\n\n> **Note:** `context_system_pct` and `context_index_pct` are parsed and validated but not yet enforced as hard caps in this release — the system prompt uses a conservative fixed limit, which is well within the configured 15% slice for all supported models. Full per-slice enforcement is planned for v1.1.\n\n### Model context window map\n\nSynthadoc ships a built-in prefix-matched table:\n\n| Model | Context window |\n|---|---|\n| `claude-opus-4*`, `claude-sonnet-4*`, `claude-haiku-4*` | 200,000 tokens |\n| `gpt-4o*`, `gpt-4-turbo*` | 128,000 tokens |\n| `gpt-4` (exact) | 8,192 tokens |\n| `gpt-3.5-turbo*` | 16,385 tokens |\n| Unknown / fallback | 128,000 tokens |\n\nSet `context_window = N` in config to override the map (useful for local Ollama models with non-standard context sizes).\n\nAt the defaults on a 200k-token model: wiki slice ≈ 480,000 chars — far more than any practical query needs. On constrained models, the budget scales down automatically so the same config works across model families.\n\n---\n\n## 32. Knowledge Graph\n\nSynthadoc computes a wikilink graph across all active and draft pages during every lint run. The graph is stored in two tables in `audit.db`:\n\n```sql\nCREATE TABLE graph_nodes (\n slug TEXT PRIMARY KEY,\n cluster_id INTEGER NOT NULL,\n updated_at TEXT NOT NULL\n);\n\nCREATE TABLE graph_edges (\n from_slug TEXT NOT NULL,\n to_slug TEXT NOT NULL,\n weight INTEGER NOT NULL DEFAULT 1,\n edge_type TEXT NOT NULL DEFAULT 'mixed',\n PRIMARY KEY (from_slug, to_slug)\n);\n```\n\nEdge weight is a composite signal from two sources:\n\n| Edge type | Source | Weight contribution |\n|-----------|--------|---------------------|\n| `wikilink` | `[[slug]]` occurrences in page body | +1 per occurrence |\n| `co_source` | pages compiled from the same source file (matched by hash) | +2 per shared source |\n\nMost edges are `mixed` (have both types). Pure `co_source` edges surface hidden relationships — pages compiled from the same source document are linked immediately after ingest, before any wikilinks are created. `edge_type` is stored on each edge so the web UI can render co-source edges with a different visual style.\n\nCluster IDs are assigned by Louvain community detection — pages that link densely to each other end up in the same cluster.\n\n### REST API\n\n**`GET /graph`** — returns the full graph with per-node enrichment:\n\n```json\n{\n \"status\": \"ready\",\n \"node_count\": 45,\n \"edge_count\": 123,\n \"cluster_count\": 4,\n \"nodes\": [\n {\"slug\": \"quantum-error-correction\", \"title\": \"Quantum Error Correction\",\n \"type\": \"concept\", \"state\": \"active\", \"cluster_id\": 2}\n ],\n \"edges\": [\n {\"from\": \"quantum-error-correction\", \"to\": \"shor-algorithm\", \"weight\": 3, \"edge_type\": \"mixed\"}\n ]\n}\n```\n\nWhen the graph tables are empty (first call after upgrade, or before the first lint run), the server returns `{\"status\": \"computing\"}` and starts a background build immediately. The web UI polls every 2 seconds and renders the graph when the status becomes `\"ready\"`.\n\n### Web UI\n\nThe **Graph** tab (alongside Chat in the top nav) renders a D3.js force-directed graph. Nodes are colored by cluster. Edge thickness is proportional to weight (1–4 px). Pure `co_source` edges (no wikilink) are rendered with a dashed stroke, making hidden source-based relationships visually distinct from explicit cross-links. Clicking a node opens a sidebar showing the page title, type and lifecycle state badges, cluster assignment, and an **\"Ask about this →\"** button that switches to the Chat tab with a pre-filled query.\n\nControls: zoom in/out (scroll or pinch), drag nodes, filter by page type.\n\n---\n\n## Customization\n\nSynthadoc exposes four extension points — all hot-loaded, no server restart required.\n\n### Custom skills (new file formats)\n\nSubclass `BaseSkill` (Apache-2.0 — no AGPL obligation on your skill code), drop the folder in `/skills/` or `~/.synthadoc/skills/`, and Synthadoc picks it up on the next ingest. Skills match by file extension or intent prefix and support any Unicode text, including CJK prefixes.\n\n→ Full interface, manifest format, and examples: [§5 Skills System](#5-skills-system) and [§17 Plugin Development Guide](#17-plugin-development-guide)\n\n### Custom LLM providers\n\nSubclass `LLMProvider` from `synthadoc/providers/base.py` (Apache-2.0) and place the file in `~/.synthadoc/providers/` or the wiki `providers/` directory. Switch active provider with one config line.\n\n→ Full interface and wiring instructions: [§10 Configuration — Provider switching](#10-configuration)\n\n### Hooks\n\nShell scripts (any language) that fire on `on_ingest_complete` and `on_lint_complete`. Receive a JSON context on stdin. Set `blocking = true` to gate the operation on the hook's exit code — useful for CI pipelines and quality gates.\n\n→ Full event schema, context JSON, and blocking behaviour: [§11 Hook System](#11-hook-system)\n\n### Cache control\n\nThree cache layers (embedding, LLM response, provider prompt cache) invalidate automatically on source-file change (SHA-256). Force a fresh call with `--force`, or wipe all cached responses with `synthadoc cache clear -w `.\n\n→ Layer-by-layer breakdown and invalidation rules: [§12 Cache System](#12-cache-system)\n\n### Multi-Platform Agent Skill Files\n\nEvery Synthadoc wiki ships three companion files that give AI coding agents a complete, self-contained operating guide for the wiki:\n\n| File | Platform |\n|---|---|\n| `AGENTS.md` | Codex CLI, OpenCode, and any agent that reads the AGENTS.md convention |\n| `CLAUDE.md` | Claude Code (this session's tool) — highest-priority instruction source in Claude's hierarchy |\n| `GEMINI.md` | Gemini CLI — relevant because Synthadoc's default LLM provider is `gemini-2.5-flash-lite` |\n\nAll three files share identical body content generated from the same template; they differ only in their H1 heading so each agent recognises its own file first. The body covers:\n\n- **Domain guidelines** — LLM-generated bullet list of domain-specific ingest and query rules (produced by `scaffold`)\n- **Quick reference table** — every common CLI command with the correct `synthadoc` syntax\n- **Server startup** — how to start the server and verify it is running\n- **Ingest examples** — local file, URL, YouTube, agent session `.jsonl`, force reingest, dry-run\n- **Query** — basic and streaming query syntax; citation marker format\n- **Lint** — when to run and what each check does\n- **Lifecycle** — transition commands with state names\n- **Page schema** — frontmatter structure with all fields\n- **MCP tools table** — all 12 tools with purpose descriptions\n\n**How the files are created and updated:**\n\n1. `synthadoc init` — writes all three files with default domain guidelines and the wiki's configured port.\n2. `synthadoc scaffold` — regenerates all three with LLM-produced domain guidelines derived from the current wiki state. The `scaffold` command prints a confirmation line for each file.\n\n**Use cases:**\n\n- Open Claude Code in a Synthadoc wiki directory and it immediately knows how to ingest, query, lint, and manage lifecycle — no additional setup. Claude Code can also start the server in background mode (`synthadoc serve -w -b`), stop it cleanly, and orchestrate multi-step workflows, effectively acting as a control plane for the wiki.\n- Gemini CLI reads `GEMINI.md` and can run `synthadoc` commands with the correct flags without asking the user for help.\n- CI pipelines with Codex CLI use `AGENTS.md` to drive automated lint and lifecycle transitions on PR merge.\n- Agent session files produced by Claude Code (`.jsonl` under `~/.claude/projects/`) can themselves be ingested into the wiki, capturing the reasoning behind design decisions alongside the source documents.\n\n→ See [README.md — Interfaces & Integration](../README.md#interfaces--integration) for the feature comparison row and quick-start instructions.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v1.1.0 (in progress)\n\n- **Weighted knowledge graph edges** — graph edges now carry two signals: wikilink occurrences (+1 each) and co-source connections — pages compiled from the same source file (matched by SHA-256 hash) — (+2 per shared source). `edge_type` field added to `graph_edges` table (`wikilink`, `co_source`, or `mixed`). `GET /graph` exposes `edge_type` per edge. Web UI renders edge thickness proportional to weight (1–4 px) and dashed lines for pure co-source edges. Co-source edges appear immediately after ingest — before any wikilinks — surfacing hidden relationships in the graph. Schema migrated to version 3.\n- **Multi-platform agent skill files** — `synthadoc init` now writes `CLAUDE.md`, `AGENTS.md`, and `GEMINI.md` at wiki root; all three carry the same complete CLI reference so Claude Code, Codex CLI, OpenCode, Pi, and Gemini CLI users get first-class guidance without manual setup.\n- **Session history ingestion skill** — `.jsonl` session files from Claude Code and Codex CLI are ingested as wiki pages; human turns and substantive assistant responses extracted; tool calls, thinking blocks, and sub-agent scaffolding skipped; format auto-detected; `suggested_slug` derived from session date and first user message.\n\n### v1.0.2\n\n- **Cascade link cleanup on archive** — archiving any page immediately removes every `[[slug]]` wikilink pointing to it from all other active pages, without waiting for the next `lint run`. Inline links become plain text; list-item-only links are dropped entirely. The HTTP `POST /lifecycle/transition` and MCP `synthadoc_lifecycle` responses now include a `cascade_links_removed_from` field listing the rewritten page slugs. During `lint run`, cascade runs once after all auto-archive transitions complete and the count is included in the lint report's `dangling_links_removed` field. System pages (index, dashboard, purpose, log, overview) and already-archived pages are skipped. The existing dangling-link lint pass remains as a safety net for dead links that predate this release.\n- **Multi-source partial-archive fix** — when a page has multiple local source files and only some are missing from disk, lint now transitions the page to `stale` (not `archived`), preserving it for re-ingest. The page is only archived when every local source file is gone. URL sources are not affected by this check.\n\n### v1.0.1\n\n- **Scaffold JSON extraction hardened** — replaced the greedy regex used to extract the LLM's JSON response with a brace-balanced parser; eliminates false-positive matches on `[[wikilink]]` syntax that previously caused `json.loads()` failures and silent scaffold corruption\n- **Generated file validation** — scaffold now validates every LLM-generated file (index, purpose, AGENTS.md) for minimum length, required frontmatter, and `[[wikilink]]` presence before writing to disk; validation failure marks the job as failed with a structured error message rather than silently writing a malformed page\n- **ROUTING.md regeneration validation** — `routing init` and scheduled routing regeneration now validate the regenerated file for minimum line count and anchor presence before applying it; prevents a corrupt or truncated ROUTING.md from entering the wiki\n- **Scaffold failure reporting** — scaffold skip and failure reasons are now surfaced in the terminal output and job error field; previously silent failures (e.g. LLM timeout, JSON parse error) are now visible without tailing the JSON log\n- **Install-time scaffold removed** — `synthadoc install` no longer runs the LLM scaffold call during installation; scaffold is a separate step (`synthadoc scaffold`) that the user runs after setting their API key, avoiding confusing API-key-not-set errors at install time\n- **AquaFlow LLM evaluation** — 15-question M&A due-diligence benchmark across five models (MiniMax-Think M3, Claude Opus 4.8, Claude Sonnet 4.6, DeepSeek-R1, Qwen Plus); full report at `docs/example/aquaflow/evaluation/report/llm-query-benchmark.md`\n\n### v1.0.0\n\n- **Pre-LLM source sanitizer** — every source document passes through a sanitizer before any LLM call; removes six categories of potentially harmful content (zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars, instruction-override phrases); a `WARN` log entry is emitted when bidi overrides or instruction-override phrases are stripped; zero LLM cost; applied before truncation so the character budget is consumed by clean text only\n- **Per-source truncation flag** — default character limit raised from 8,000 to 32,000 characters per source (~8,000 tokens); truncated sources are flagged with `truncated: true` in the page's `sources:` frontmatter; `--max-source-chars N` CLI flag and matching `POST /jobs/ingest` body field and MCP `synthadoc_ingest` parameter override the limit per run; `synthadoc lint` emits a warning for any page with a truncated source and suggests the override command; configurable via `[ingest] max_source_chars` in `config.toml`\n- **Proportional context budget** — query context is allocated proportionally to the configured model's context window; four configurable slices: `context_wiki_pct` (60%), `context_history_pct` (20%), `context_system_pct` (15%), `context_index_pct` (5%); built-in prefix-matched model context window table covers Claude 4, GPT-4o/turbo/3.5, and GPT-4; unknown models fall back to 128,000 tokens; `context_window = N` in `[query]` overrides the table for local or custom models\n- **Knowledge graph** — wikilink graph computed across all active and draft pages during every lint run; stored in `graph_nodes` and `graph_edges` tables in `audit.db`; edge weight = number of `[[wikilink]]` references between two pages; cluster IDs assigned by Louvain community detection; `GET /graph` REST endpoint returns nodes with per-node enrichment (title, type, state, cluster) and edges; first call after upgrade triggers a background build and returns `{\"status\":\"computing\"}`; web UI **Graph** tab (D3.js force-directed graph) — nodes colored by cluster, click to see page details and an \"Ask about this →\" button that opens a pre-filled chat query\n- **Citation accuracy & rendering improvements** — six correctness fixes to the ingest citation pipeline: (1) full source text passed to the decision LLM so citations can reference any line in the document; (2) `_annotate_citations` bugs A–F resolved (false-positive matches, off-by-one line ranges, duplicate markers, escaped bracket handling, multi-paragraph span errors, and source filename normalisation); (3) `_has_citations` check added — lint warns and suggests a model upgrade when a page has zero citation markers after synthesis; (4) `Key Data` section extraction hardened so numerical facts, rates, and formulas are preserved verbatim across the annotation pass; (5) `obsidianCitationsToGfm` rendering fixed in the web UI — citation superscripts now display correctly in the chat panel alongside streamed answers; (6) reading view set as the default Obsidian display mode for wiki pages\n\n### v0.9.3\n\n- **Backup now includes `hooks/` and `.synthadoc/extracted/`** — both directories were silently omitted from backups. `hooks/` contains user-customised hook scripts; `.synthadoc/extracted/` contains per-source text sidecars and PDF pagemaps written by the ingest agent. Without these, the Source Viewer and Provenance citation modal showed \"Could not read source file\" after restore.\n- **PyPI publish workflow** — `publish.yml` triggers on GitHub Release, builds the wheel with hatchling, and uploads to PyPI via OIDC trusted publishing (no API token stored). `pip install synthadoc` now works.\n- **Web UI bundled in pip package** — `synthadoc/data/web-ui/dist/` is now included in the wheel. `synthadoc web` previously returned 503 on pip-installed instances because the React dist was not packaged.\n- **CI sync checks** — PRs now fail if `synthadoc/data/obsidian-plugin/`, `synthadoc/data/web-ui/`, or `README-pypi.md` are out of sync with their source. All sync-check steps use `shell: bash` for Windows runner compatibility.\n- **`bump_version.py` extended** — now also updates `synthadoc/data/obsidian-plugin/manifest.json` alongside the source manifest, preventing version drift between the bundled copy and the plugin source.\n\n### v0.9.2\n\n- **`synthadoc/data/web-ui/dist/` packaged** — web UI dist files committed to the repo and bundled in the wheel via hatchling artifacts.\n- **`scripts/sync_web_ui.py`** — helper script to sync `web-ui/dist/` into `synthadoc/data/web-ui/dist/` after `npm run build`.\n\n### v0.9.1\n\n- **PyPI metadata** — added `readme`, `keywords`, `classifiers`, and `[project.urls]` to `pyproject.toml` for correct PyPI rendering.\n- **`README-pypi.md`** — generated from `README.md` by `scripts/prepare_pypi_readme.py`; strips `pypi-strip` blocks and rewrites relative links to absolute GitHub URLs.\n- **Branch protection compatibility** — removed post-merge `build-plugin` CI job that pushed compiled plugin directly to `main`. Plugin sync is now enforced in PRs: CI fails if `synthadoc/data/obsidian-plugin/` is out of sync.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 5). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n"}, "24abd7e5ac8ccb1a5288c300eba8d652d45ba35d": {"t": 1785767258.674569, "n": 204422, "h": "7dc44194893f6dbe", "c": "# Synthadoc — Design Document\n\n**Version:** 1.0.2 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n29. [Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer)\n30. [Per-Source Truncation Flag](#30-per-source-truncation-flag)\n31. [Proportional Context Budget](#31-proportional-context-budget)\n32. [Knowledge Graph](#32-knowledge-graph)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the full source text (bounded by `max_source_chars`) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `flag` / `skip`). Pages with `status='active'` are protected — sources that conflict with them trigger `flag` rather than `update` (RULE 1b).\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Orchestrator calls `_guard_ingest_cost()` on the returned cost: soft warn → `logger.warning` only; hard gate → `cost_gate_exceeded` audit event + `fail_permanent` (job → DEAD). On success, job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 0b — Key Data Extraction | None (regex) | Deterministic pre-processor: extracts numbers, percentages, currency amounts, ratios, formulas, underscore identifiers, and date ranges from source text via regex; appends them as a `[Key Data — extracted by pre-processor]` section before any LLM sees the text. Zero token cost; cannot hallucinate. Anchors the synthesis LLM to all quantitative candidates. |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads the full source text (bounded by `max_source_chars`) + BM25 candidates + `purpose.md` scope + status of each candidate page. Outputs per-page action: `create`, `update`, `flag`, `skip`. **RULE 1b:** a page with `status='active'` is human-reviewed and authoritative — when the source provides a conflicting value, date, formula, or conclusion, the action must be `flag`, not `update`. The only exception is a source that adds a topic the page does not yet mention at all. Decision cache version: `v2`. |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. If no citation markers are present in the returned body, a `citation_pass4_no_markers` WARNING audit event is written — the page is still saved but the condition is flagged for re-ingest with a more capable model. Zero-citation results are not cached so a subsequent re-ingest can succeed. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic. **Suppressed when TF fallback was used** (a small corpus with broad coverage may legitimately return fewer than 3 candidates — that is expected, not a gap).\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n| Citation presence (Check 5b) | Page body has ≥ 50 words and zero `^[filename:L-L]` citation markers → WARNING. Diagnostic only — does not block promotion to `active`. Indicates the annotation pass failed to produce markers, usually a model-compatibility issue (use Gemini 2.5 Flash or higher for reliable citation annotation). |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n### ActionAgent\n\nDispatches action-intent queries from the chat UI (e.g. \"activate a draft page\", \"show lint report\", \"archive a stale page\", \"run scaffold\"). Parses the user's free-text intent into a structured action via an LLM extraction prompt, then executes the action against the wiki and returns a human-readable result. When the intent is ambiguous (e.g. \"activate a page\" without specifying which), sets `needs_clarification=True` and returns a prompt and candidate list for the web UI to render as chip buttons.\n\n### ScaffoldAgent\n\nGenerates and updates the wiki's structural pages. Reads all current wiki pages and asks the LLM to organise them into 5–8 domain-appropriate categories, then writes `wiki/index.md` (category index with wikilinks), `wiki/purpose.md` (scope statement), and `AGENTS.md` (ingest and query guidelines). Also stamps a `categories:` field on each page's YAML frontmatter. When `ROUTING.md` already exists, regenerates it to stay in sync with the updated index structure.\n\n### RewriteAgent\n\nRewrites follow-up questions in multi-turn conversations into self-contained, standalone form before BM25 retrieval. Converts context-dependent phrases (\"What came after that?\", \"Tell me more about his early life\") into explicit questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the correct pages without relying on conversation context. Only invoked when conversation history is non-empty. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### SummarizeAgent\n\nCompresses the oldest conversation turns into a single `[Session summary: …]` assistant turn when a session exceeds `conversation_history_turns`. Prevents unbounded context growth across long sessions. Emits a `notice` SSE event the first time compression occurs so the user can see that earlier context was condensed. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### ContextAgent\n\nBuilds token-budgeted context packs for MCP tool consumers. See [Section 20 — Context Packs](#20-context-packs) for full detail.\n\n### SearchDecomposeAgent\n\nDecomposes a web search intent into 1–4 terse keyword search strings optimised for authoritative source retrieval. Uses a separate prompt from `QueryAgent`'s question decomposition — search decomposition asks \"what distinct search strings would find the best sources?\" (terse keyword phrases) rather than \"what distinct questions does this ask?\" (natural-language sub-questions). See [Web search fan-out](#web-search-fan-out) in the IngestAgent section.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n| `session` | `.jsonl` | `claude session`, `codex session`, `cursor session`, `ai session`, `session history` | Extracts human-readable turns from AI coding session transcripts. Supports Claude Code JSONL format and Codex/Cursor format. Filters tool calls, thinking blocks, and sub-agent scaffolding. Short turns (< 20 assistant words, < 3 user words) are skipped. No external dependencies. |\n\n### Session Skill — AI Session History Ingestion\n\nThe `session` skill turns AI coding session history files (`.jsonl`) into searchable wiki pages capturing the problem-solving knowledge exchanged during the session.\n\n**Supported formats**\n\n| Format | File origin | Detection |\n|--------|-------------|-----------|\n| Claude Code | `~/.claude/projects//.jsonl` | `obj.type` ∈ `{\"user\", \"assistant\"}` with nested `obj.message` |\n| Codex / Cursor | Export from OpenAI Codex or Cursor IDE | `obj.role` at top level, no `message` wrapper |\n\nFormat is auto-detected from the first 30 parseable lines. Files that match neither format are parsed as Codex (fallback).\n\n**Filtering rules**\n\n| Content | Kept? | Reason |\n|---------|-------|--------|\n| User text ≥ 3 words | ✓ | Substantive question or instruction |\n| Assistant text ≥ 20 words | ✓ | Substantive answer |\n| User text < 3 words (`\"ok\"`, `\"yes\"`) | ✗ | Too terse to be useful |\n| Assistant text < 20 words | ✗ | Acknowledgement or one-liner |\n| `isSidechain: true` lines | ✗ | Sub-agent scaffolding (not the user's conversation) |\n| `tool_use` blocks | ✗ | Shell commands / file writes — not final output |\n| `tool_result` blocks | ✗ | Raw output — avoids leaking file contents or credentials |\n| `thinking` blocks | ✗ | Internal reasoning — not the final answer |\n| `permission-mode`, `system`, `last-prompt` lines | ✗ | Session metadata, not conversation |\n\nAfter extraction the text passes through Synthadoc's standard pre-LLM source sanitizer\n(zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars,\ninstruction-override phrases) — the same step applied to every PDF, DOCX, URL, and other source type.\nSee [§29 Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer).\n\n**Output format**\n\nEach kept turn is labelled and separated by `---`:\n\n```\n[USER]\nHow do I implement a sliding window algorithm in Python?\n\n---\n\n[ASSISTANT]\nA sliding window algorithm maintains a contiguous subarray by advancing two\npointers simultaneously…\n```\n\n**`suggested_slug`**\n\nThe skill sets `metadata[\"suggested_slug\"]` to `session-YYYY-MM-DD-` where the date comes from the file's `mtime` and the topic from the first substantive user turn (first 6 words, slugified). Example: `session-2026-07-15-how-do-i-implement-sliding`.\n\n**Limitations**\n\n- No chunking in v1.1 — very long sessions are truncated at `max_source_chars` (default 400 000 chars). Split large archives into individual `.jsonl` files.\n- Tool output is excluded by design — re-ingest the original source files if you need the file contents in the wiki.\n- Re-ingesting the same `.jsonl` file uses Synthadoc's standard source-hash dedup — no duplicate pages are created.\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n**Compound identifier expansion:** When a token contains underscores, the tokenizer emits both the compound form and each component as separate tokens (`capex_growth` → `[\"capex_growth\", \"capex\", \"growth\"]`). This expansion is applied at both index time and query time so that human-friendly queries match programmatic identifiers and vice versa. Leading/trailing underscores are stripped before splitting; no empty strings are emitted.\n\n**TF fallback for small corpora:** When all BM25 scores in a result set are zero or negative (IDF collapse on a corpus of ≤ 2 documents, or a term present in every page), the search system falls back to normalised term frequency (TF) scoring: `TF(doc, query) = Σcount(term, doc) / len(doc_tokens)`. Results are returned in the same format as BM25 results with a `tf_fallback: True` flag on each `SearchResult`. This field is recorded in the audit trail and consumed by gap detection (see below).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp, cascade_links_removed_from: [str]}` _(cascade field added v1.0.2)_ |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n| `GET` | `/graph` _(v1.0.0)_ | — | `{status, node_count, edge_count, cluster_count, nodes: [...], edges: [...]}` or `{status: \"computing\"}` on first call |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n**External file paths** (outside the wiki root) are supported when the request comes from\n`127.0.0.1` or `::1` and the payload includes `\"allow_external_paths\": true`. The CLI sets\nthis flag automatically for all local file sources. Remote clients cannot set this flag\n(the server ignores it for non-localhost connections) to prevent arbitrary file reads\nin server-exposed deployments.\n\nThis is required for ingesting files that live outside the wiki directory by design —\nfor example, Claude Code session transcripts at `~/.claude/projects//.jsonl`.\n\n**Client behaviour summary:**\n\n| Client | Sends `allow_external_paths` | Can ingest outside wiki root? |\n|--------|------------------------------|-------------------------------|\n| CLI (`synthadoc ingest `) | `true` (automatic, local file paths only) | Yes — server is localhost |\n| Obsidian plugin | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Web UI | Never sent (field omitted → defaults `false`) | No — wiki-relative paths and URLs only |\n| Direct HTTP API (localhost) | Set manually in request body | Yes — if request from 127.0.0.1/::1 |\n| Direct HTTP API (remote) | Ignored by server | No — server silently treats as `false` |\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 1.0.2\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n**Reading View default:** `synthadoc plugin install` and `synthadoc plugin upgrade` write `\"defaultViewMode\": \"preview\"` to `.obsidian/app.json` after copying the plugin files. This causes Obsidian to open new notes in Reading View by default, which is required for `^[filename:L-L]` citation chips to render. Only `defaultViewMode` is written; all other keys in `app.json` are preserved. If `app.json` is malformed JSON the write is skipped and a warning is logged — the install still succeeds.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports two things: dangling slugs (pages listed in ROUTING.md that no longer exist in the wiki) and unassigned slugs (pages that exist in index.md but are not listed in any ROUTING.md branch). Enabled only when ROUTING.md exists. **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [-w wiki] — generate ROUTING.md from index.md branch structure\n│ ├── validate [-w wiki] — report dangling slugs and cross-branch duplicates\n│ └── clean [-w wiki] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [-w wiki]\n├── candidates\n│ ├── list [-w wiki]\n│ ├── promote |--all [-w wiki]\n│ └── discard |--all [-w wiki]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n# Citation pass (Pass 4) tuning — these two settings work together:\n# citation_source_lines — how many lines of the source the LLM sees when placing ^[...] markers.\n# Increase if lint reports out_of_range on long sources (transcripts, PDFs).\n# citation_max_tokens — output token budget for the annotated section returned by the LLM.\n# Increase if you raise citation_source_lines and have long wiki sections.\n# citation_source_lines = 400\n# citation_max_tokens = 8192\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Emit `logger.warning` in server log when per-job ingest cost exceeds this threshold; job continues normally |\n| `cost.hard_gate_usd` | float | `2.00` | Permanently fail the ingest job (DEAD) and record a `cost_gate_exceeded` audit event when per-job cost exceeds this threshold |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × (backup_count + 1)` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n| `ingest.max_source_chars` | int | `32000` | Character limit applied to each source before the LLM call. Sources exceeding this limit are truncated; the page's `sources:` frontmatter entry gets `truncated: true` and lint emits a warning. Override per-run with `--max-source-chars N`. _(v1.0.0)_ |\n| `ingest.citation_source_lines` | int | `400` | Number of source lines the LLM sees during Pass 4 (citation annotation). Increase if lint reports `out_of_range` on long sources such as transcripts or PDFs. Raise `citation_max_tokens` in proportion when increasing this value. |\n| `ingest.citation_max_tokens` | int | `8192` | Output token budget for the annotated section returned by Pass 4. Increase when `citation_source_lines` is raised or wiki sections are long, to avoid truncated annotations. |\n| `query.context_wiki_pct` | float | `0.60` | Fraction of the model context window reserved for wiki source pages. _(v1.0.0)_ |\n| `query.context_history_pct` | float | `0.20` | Fraction reserved for conversation history. _(v1.0.0)_ |\n| `query.context_system_pct` | float | `0.15` | Fraction reserved for system prompt and instructions. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_index_pct` | float | `0.05` | Fraction reserved for the wiki index preamble. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_window` | int | _(auto)_ | Override the built-in context window lookup for the configured model. Use when running a local or custom model whose window size is not in the built-in table. _(v1.0.0)_ |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement is wired into the ingest pipeline: `Orchestrator._guard_ingest_cost()` is called after each ingest LLM call with the returned cost, before the job is marked complete.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | `logger.warning` in server log; job continues normally |\n| `hard_gate_usd` | $2.00 | Server/ingest path: records `cost_gate_exceeded` audit event + permanently fails job (DEAD). Interactive CLI: prompts `Proceed? [y/N]` |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` by default. The bind address is configurable via `server.host` in `config.toml` (e.g. `host = \"0.0.0.0\"` to expose on all interfaces for LAN access). No built-in authentication — restrict via firewall when exposing beyond loopback. Remote access without a reverse proxy is not recommended on shared or networked machines.\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs and unassigned slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `-w ` / `--wiki ` to target a specific wiki.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; when all local source files are missing from disk the page is transitioned to `archived` (if only some sources are missing it is transitioned to `stale` instead). Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: lint evaluates all local (non-URL) source paths listed in the page's frontmatter. If **all** of them are missing from disk, the page is transitioned to `archived`. If only **some** are missing — meaning valid content from at least one source still exists — the page is transitioned to `stale` instead, preserving it for re-ingest rather than discarding it. _(v1.0.2: previously any single missing source triggered an immediate archive.)_\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Cascade link cleanup\n\nWhenever a page transitions to `archived` — whether triggered manually (`synthadoc lifecycle archive`), via the Obsidian Lifecycle modal, the MCP `synthadoc_lifecycle` tool, or automatically by lint detecting a missing source — Synthadoc immediately scans all other active pages and removes every `[[archived-slug]]` wikilink pointing to the now-archived page.\n\n- **Inline links** — `[[archived-slug]]` becomes plain text (the display text if an alias was used, otherwise the slug itself).\n- **List-item links** — a list item whose only content is `[[archived-slug]]` is dropped entirely.\n- **Archived pages are skipped** — pages already archived are not rewritten.\n- **System pages are skipped** — index, overview, dashboard, log, and purpose pages are excluded.\n\nThe cleanup runs synchronously before the response is returned, so the wiki is consistent immediately. The CLI and MCP responses include a `cascade_links_removed_from` field listing the affected slugs. During `lint run`, cascade is batched once after all auto-archive transitions complete, and the count is added to `dangling_links_removed` in the lint report.\n\nLint's existing periodic dangling-link cleanup (`lint run --scope orphans`) remains in place as a safety net for any dead links that predate this feature or arrive through other paths.\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 5) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 5 # turns to include in each request (default: 5; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp, cascade_links_removed_from: [str]}` or `{error, cascade_links_removed_from: []}` _(cascade field added v1.0.2)_ | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── hooks/ ← user hook scripts (always included)\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ ├── extracted/ ← text sidecars + PDF pagemaps (always included)\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `logs/`.\n\n> **v0.9.3 fix:** `hooks/` and `.synthadoc/extracted/` were previously missing from backups. Omitting `extracted/` caused the Source Viewer and Provenance citation modal to show \"Could not read source file\" after restore.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## 29. Pre-LLM Source Sanitizer\n\nEvery source document passes through a sanitizer immediately after text extraction, before any LLM call. The sanitizer removes six categories of content that could manipulate the LLM or degrade compilation quality:\n\n| Category | Action | Warning logged? |\n|---|---|---|\n| Zero-width characters (U+200B, U+200C, U+200D, U+FEFF) | Removed silently | No |\n| Bidi override characters (U+202A–U+202E, U+2066–U+2069) | Removed | Yes |\n| HTML comments (``) | Removed silently | No |\n| Hidden CSS spans (`display:none`, `visibility:hidden`) | Removed silently | No |\n| Base64 blobs ≥ 200 consecutive characters | Replaced with `[base64 content removed]` | Yes |\n| Instruction-override phrases (8 patterns: \"ignore previous instructions\", \"disregard the above\", \"override your system prompt\", etc.) | Replaced with `[redacted]` | Always |\n\nWhen content is removed and a warning is appropriate, the server logs a `WARN` entry:\n```\n[WARN] sanitizer stripped content from 'papers/survey.pdf': bidi overrides, instruction-override phrase\n```\n\nThis is a pure pre-processing step with zero LLM cost. Truncation (if configured) is applied after sanitization, so the character budget is consumed by clean text only.\n\n---\n\n## 30. Per-Source Truncation Flag\n\nLarge source documents are truncated before they reach the LLM. By default the limit is **32,000 characters** (~8,000 tokens), raised from the prior hardcoded 8,000-character limit. When a source exceeds this limit, its compiled wiki page records a `truncated: true` flag in its `sources:` frontmatter entry.\n\n### Configuration\n\n```toml\n[ingest]\nmax_source_chars = 32000 # default; raise for large PDFs or books\n```\n\n### Per-ingest override\n\nThe limit can be raised for a single ingest run without modifying the config:\n\n```bash\nsynthadoc ingest papers/large-textbook.pdf --max-source-chars 128000\n```\n\nThe same `max_source_chars` field is accepted by the HTTP `POST /jobs/ingest` body and the MCP `synthadoc_ingest` tool.\n\n### Lint surfacing\n\n`synthadoc lint` emits a warning for any page that has a truncated source:\n\n```\n[WARN] quantum-computing.md: source 'papers/large-survey.pdf' was truncated at ingest\n (source exceeded max_source_chars=32000 — 87,412 chars in source).\n To re-ingest with a higher limit (this source only):\n synthadoc ingest papers/large-survey.pdf --max-source-chars 128000\n To raise the limit for all future ingests:\n set [ingest] max_source_chars = 128000 in your config\n```\n\n---\n\n## 31. Proportional Context Budget\n\nQuery context is allocated proportionally to the configured model's context window, replacing the prior hard cap. The default allocation:\n\n| Slice | Default | Purpose |\n|---|---|---|\n| `context_wiki_pct` | 60% | Wiki page content (ranked by BM25 + vector score) |\n| `context_history_pct` | 20% | Chat turn history (newest-first, oldest dropped when over budget) |\n| `context_system_pct` | 15% | System prompt and purpose document |\n| `context_index_pct` | 5% | `ROUTING.md` and search index |\n\nPercentages must sum to ≤ 100. Configure in `[query]`:\n\n```toml\n[query]\ncontext_window = 0 # 0 = auto-detect from known model map\ncontext_wiki_pct = 60\ncontext_history_pct = 20\ncontext_system_pct = 15\ncontext_index_pct = 5\n```\n\n> **Note:** `context_system_pct` and `context_index_pct` are parsed and validated but not yet enforced as hard caps in this release — the system prompt uses a conservative fixed limit, which is well within the configured 15% slice for all supported models. Full per-slice enforcement is planned for v1.1.\n\n### Model context window map\n\nSynthadoc ships a built-in prefix-matched table:\n\n| Model | Context window |\n|---|---|\n| `claude-opus-4*`, `claude-sonnet-4*`, `claude-haiku-4*` | 200,000 tokens |\n| `gpt-4o*`, `gpt-4-turbo*` | 128,000 tokens |\n| `gpt-4` (exact) | 8,192 tokens |\n| `gpt-3.5-turbo*` | 16,385 tokens |\n| Unknown / fallback | 128,000 tokens |\n\nSet `context_window = N` in config to override the map (useful for local Ollama models with non-standard context sizes).\n\nAt the defaults on a 200k-token model: wiki slice ≈ 480,000 chars — far more than any practical query needs. On constrained models, the budget scales down automatically so the same config works across model families.\n\n---\n\n## 32. Knowledge Graph\n\nSynthadoc computes a wikilink graph across all active and draft pages during every lint run. The graph is stored in two tables in `audit.db`:\n\n```sql\nCREATE TABLE graph_nodes (\n slug TEXT PRIMARY KEY,\n cluster_id INTEGER NOT NULL,\n updated_at TEXT NOT NULL\n);\n\nCREATE TABLE graph_edges (\n from_slug TEXT NOT NULL,\n to_slug TEXT NOT NULL,\n weight INTEGER NOT NULL DEFAULT 1,\n PRIMARY KEY (from_slug, to_slug)\n);\n```\n\nEdge weight equals the number of `[[wikilink]]` references from one page to another (most pairs: 1). Cluster IDs are assigned by Louvain community detection — pages that link densely to each other end up in the same cluster.\n\n### REST API\n\n**`GET /graph`** — returns the full graph with per-node enrichment:\n\n```json\n{\n \"status\": \"ready\",\n \"node_count\": 45,\n \"edge_count\": 123,\n \"cluster_count\": 4,\n \"nodes\": [\n {\"slug\": \"quantum-error-correction\", \"title\": \"Quantum Error Correction\",\n \"type\": \"concept\", \"state\": \"active\", \"cluster_id\": 2}\n ],\n \"edges\": [\n {\"from\": \"quantum-error-correction\", \"to\": \"shor-algorithm\", \"weight\": 1}\n ]\n}\n```\n\nWhen the graph tables are empty (first call after upgrade, or before the first lint run), the server returns `{\"status\": \"computing\"}` and starts a background build immediately. The web UI polls every 2 seconds and renders the graph when the status becomes `\"ready\"`.\n\n### Web UI\n\nThe **Graph** tab (alongside Chat in the top nav) renders a D3.js force-directed graph. Nodes are colored by cluster. Clicking a node opens a sidebar showing the page title, type and lifecycle state badges, cluster assignment, and an **\"Ask about this →\"** button that switches to the Chat tab with a pre-filled query.\n\nControls: zoom in/out (scroll or pinch), drag nodes, filter by page type.\n\n---\n\n## Customization\n\nSynthadoc exposes four extension points — all hot-loaded, no server restart required.\n\n### Custom skills (new file formats)\n\nSubclass `BaseSkill` (Apache-2.0 — no AGPL obligation on your skill code), drop the folder in `/skills/` or `~/.synthadoc/skills/`, and Synthadoc picks it up on the next ingest. Skills match by file extension or intent prefix and support any Unicode text, including CJK prefixes.\n\n→ Full interface, manifest format, and examples: [§5 Skills System](#5-skills-system) and [§17 Plugin Development Guide](#17-plugin-development-guide)\n\n### Custom LLM providers\n\nSubclass `LLMProvider` from `synthadoc/providers/base.py` (Apache-2.0) and place the file in `~/.synthadoc/providers/` or the wiki `providers/` directory. Switch active provider with one config line.\n\n→ Full interface and wiring instructions: [§10 Configuration — Provider switching](#10-configuration)\n\n### Hooks\n\nShell scripts (any language) that fire on `on_ingest_complete` and `on_lint_complete`. Receive a JSON context on stdin. Set `blocking = true` to gate the operation on the hook's exit code — useful for CI pipelines and quality gates.\n\n→ Full event schema, context JSON, and blocking behaviour: [§11 Hook System](#11-hook-system)\n\n### Cache control\n\nThree cache layers (embedding, LLM response, provider prompt cache) invalidate automatically on source-file change (SHA-256). Force a fresh call with `--force`, or wipe all cached responses with `synthadoc cache clear -w `.\n\n→ Layer-by-layer breakdown and invalidation rules: [§12 Cache System](#12-cache-system)\n\n### Multi-Platform Agent Skill Files\n\nEvery Synthadoc wiki ships three companion files that give AI coding agents a complete, self-contained operating guide for the wiki:\n\n| File | Platform |\n|---|---|\n| `AGENTS.md` | Codex CLI, OpenCode, and any agent that reads the AGENTS.md convention |\n| `CLAUDE.md` | Claude Code (this session's tool) — highest-priority instruction source in Claude's hierarchy |\n| `GEMINI.md` | Gemini CLI — relevant because Synthadoc's default LLM provider is `gemini-2.5-flash-lite` |\n\nAll three files share identical body content generated from the same template; they differ only in their H1 heading so each agent recognises its own file first. The body covers:\n\n- **Domain guidelines** — LLM-generated bullet list of domain-specific ingest and query rules (produced by `scaffold`)\n- **Quick reference table** — every common CLI command with the correct `synthadoc` syntax\n- **Server startup** — how to start the server and verify it is running\n- **Ingest examples** — local file, URL, YouTube, agent session `.jsonl`, force reingest, dry-run\n- **Query** — basic and streaming query syntax; citation marker format\n- **Lint** — when to run and what each check does\n- **Lifecycle** — transition commands with state names\n- **Page schema** — frontmatter structure with all fields\n- **MCP tools table** — all 12 tools with purpose descriptions\n\n**How the files are created and updated:**\n\n1. `synthadoc init` — writes all three files with default domain guidelines and the wiki's configured port.\n2. `synthadoc scaffold` — regenerates all three with LLM-produced domain guidelines derived from the current wiki state. The `scaffold` command prints a confirmation line for each file.\n\n**Use cases:**\n\n- Open Claude Code in a Synthadoc wiki directory and it immediately knows how to ingest, query, lint, and manage lifecycle — no additional setup. Claude Code can also start the server in background mode (`synthadoc serve -w -b`), stop it cleanly, and orchestrate multi-step workflows, effectively acting as a control plane for the wiki.\n- Gemini CLI reads `GEMINI.md` and can run `synthadoc` commands with the correct flags without asking the user for help.\n- CI pipelines with Codex CLI use `AGENTS.md` to drive automated lint and lifecycle transitions on PR merge.\n- Agent session files produced by Claude Code (`.jsonl` under `~/.claude/projects/`) can themselves be ingested into the wiki, capturing the reasoning behind design decisions alongside the source documents.\n\n→ See [README.md — Interfaces & Integration](../README.md#interfaces--integration) for the feature comparison row and quick-start instructions.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v1.0.2\n\n- **Cascade link cleanup on archive** — archiving any page immediately removes every `[[slug]]` wikilink pointing to it from all other active pages, without waiting for the next `lint run`. Inline links become plain text; list-item-only links are dropped entirely. The HTTP `POST /lifecycle/transition` and MCP `synthadoc_lifecycle` responses now include a `cascade_links_removed_from` field listing the rewritten page slugs. During `lint run`, cascade runs once after all auto-archive transitions complete and the count is included in the lint report's `dangling_links_removed` field. System pages (index, dashboard, purpose, log, overview) and already-archived pages are skipped. The existing dangling-link lint pass remains as a safety net for dead links that predate this release.\n- **Multi-source partial-archive fix** — when a page has multiple local source files and only some are missing from disk, lint now transitions the page to `stale` (not `archived`), preserving it for re-ingest. The page is only archived when every local source file is gone. URL sources are not affected by this check.\n\n### v1.0.1\n\n- **Scaffold JSON extraction hardened** — replaced the greedy regex used to extract the LLM's JSON response with a brace-balanced parser; eliminates false-positive matches on `[[wikilink]]` syntax that previously caused `json.loads()` failures and silent scaffold corruption\n- **Generated file validation** — scaffold now validates every LLM-generated file (index, purpose, AGENTS.md) for minimum length, required frontmatter, and `[[wikilink]]` presence before writing to disk; validation failure marks the job as failed with a structured error message rather than silently writing a malformed page\n- **ROUTING.md regeneration validation** — `routing init` and scheduled routing regeneration now validate the regenerated file for minimum line count and anchor presence before applying it; prevents a corrupt or truncated ROUTING.md from entering the wiki\n- **Scaffold failure reporting** — scaffold skip and failure reasons are now surfaced in the terminal output and job error field; previously silent failures (e.g. LLM timeout, JSON parse error) are now visible without tailing the JSON log\n- **Install-time scaffold removed** — `synthadoc install` no longer runs the LLM scaffold call during installation; scaffold is a separate step (`synthadoc scaffold`) that the user runs after setting their API key, avoiding confusing API-key-not-set errors at install time\n- **AquaFlow LLM evaluation** — 15-question M&A due-diligence benchmark across five models (MiniMax-Think M3, Claude Opus 4.8, Claude Sonnet 4.6, DeepSeek-R1, Qwen Plus); full report at `docs/example/aquaflow/evaluation/report/llm-query-benchmark.md`\n\n### v1.0.0\n\n- **Pre-LLM source sanitizer** — every source document passes through a sanitizer before any LLM call; removes six categories of potentially harmful content (zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars, instruction-override phrases); a `WARN` log entry is emitted when bidi overrides or instruction-override phrases are stripped; zero LLM cost; applied before truncation so the character budget is consumed by clean text only\n- **Per-source truncation flag** — default character limit raised from 8,000 to 32,000 characters per source (~8,000 tokens); truncated sources are flagged with `truncated: true` in the page's `sources:` frontmatter; `--max-source-chars N` CLI flag and matching `POST /jobs/ingest` body field and MCP `synthadoc_ingest` parameter override the limit per run; `synthadoc lint` emits a warning for any page with a truncated source and suggests the override command; configurable via `[ingest] max_source_chars` in `config.toml`\n- **Proportional context budget** — query context is allocated proportionally to the configured model's context window; four configurable slices: `context_wiki_pct` (60%), `context_history_pct` (20%), `context_system_pct` (15%), `context_index_pct` (5%); built-in prefix-matched model context window table covers Claude 4, GPT-4o/turbo/3.5, and GPT-4; unknown models fall back to 128,000 tokens; `context_window = N` in `[query]` overrides the table for local or custom models\n- **Knowledge graph** — wikilink graph computed across all active and draft pages during every lint run; stored in `graph_nodes` and `graph_edges` tables in `audit.db`; edge weight = number of `[[wikilink]]` references between two pages; cluster IDs assigned by Louvain community detection; `GET /graph` REST endpoint returns nodes with per-node enrichment (title, type, state, cluster) and edges; first call after upgrade triggers a background build and returns `{\"status\":\"computing\"}`; web UI **Graph** tab (D3.js force-directed graph) — nodes colored by cluster, click to see page details and an \"Ask about this →\" button that opens a pre-filled chat query\n- **Citation accuracy & rendering improvements** — six correctness fixes to the ingest citation pipeline: (1) full source text passed to the decision LLM so citations can reference any line in the document; (2) `_annotate_citations` bugs A–F resolved (false-positive matches, off-by-one line ranges, duplicate markers, escaped bracket handling, multi-paragraph span errors, and source filename normalisation); (3) `_has_citations` check added — lint warns and suggests a model upgrade when a page has zero citation markers after synthesis; (4) `Key Data` section extraction hardened so numerical facts, rates, and formulas are preserved verbatim across the annotation pass; (5) `obsidianCitationsToGfm` rendering fixed in the web UI — citation superscripts now display correctly in the chat panel alongside streamed answers; (6) reading view set as the default Obsidian display mode for wiki pages\n\n### v0.9.3\n\n- **Backup now includes `hooks/` and `.synthadoc/extracted/`** — both directories were silently omitted from backups. `hooks/` contains user-customised hook scripts; `.synthadoc/extracted/` contains per-source text sidecars and PDF pagemaps written by the ingest agent. Without these, the Source Viewer and Provenance citation modal showed \"Could not read source file\" after restore.\n- **PyPI publish workflow** — `publish.yml` triggers on GitHub Release, builds the wheel with hatchling, and uploads to PyPI via OIDC trusted publishing (no API token stored). `pip install synthadoc` now works.\n- **Web UI bundled in pip package** — `synthadoc/data/web-ui/dist/` is now included in the wheel. `synthadoc web` previously returned 503 on pip-installed instances because the React dist was not packaged.\n- **CI sync checks** — PRs now fail if `synthadoc/data/obsidian-plugin/`, `synthadoc/data/web-ui/`, or `README-pypi.md` are out of sync with their source. All sync-check steps use `shell: bash` for Windows runner compatibility.\n- **`bump_version.py` extended** — now also updates `synthadoc/data/obsidian-plugin/manifest.json` alongside the source manifest, preventing version drift between the bundled copy and the plugin source.\n\n### v0.9.2\n\n- **`synthadoc/data/web-ui/dist/` packaged** — web UI dist files committed to the repo and bundled in the wheel via hatchling artifacts.\n- **`scripts/sync_web_ui.py`** — helper script to sync `web-ui/dist/` into `synthadoc/data/web-ui/dist/` after `npm run build`.\n\n### v0.9.1\n\n- **PyPI metadata** — added `readme`, `keywords`, `classifiers`, and `[project.urls]` to `pyproject.toml` for correct PyPI rendering.\n- **`README-pypi.md`** — generated from `README.md` by `scripts/prepare_pypi_readme.py`; strips `pypi-strip` blocks and rewrites relative links to absolute GitHub URLs.\n- **Branch protection compatibility** — removed post-merge `build-plugin` CI job that pushed compiled plugin directly to `main`. Plugin sync is now enforced in PRs: CI fails if `synthadoc/data/obsidian-plugin/` is out of sync.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 5). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n"}, "f1cee3dc9475c3ce0496f22582599c77d5eba38b": {"t": 1785767260.3743272, "n": 200042, "h": "239ef8c9f4036384", "c": "# Synthadoc — Design Document\n\n**Version:** 1.0.2 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n29. [Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer)\n30. [Per-Source Truncation Flag](#30-per-source-truncation-flag)\n31. [Proportional Context Budget](#31-proportional-context-budget)\n32. [Knowledge Graph](#32-knowledge-graph)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the full source text (bounded by `max_source_chars`) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `flag` / `skip`). Pages with `status='active'` are protected — sources that conflict with them trigger `flag` rather than `update` (RULE 1b).\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Orchestrator calls `_guard_ingest_cost()` on the returned cost: soft warn → `logger.warning` only; hard gate → `cost_gate_exceeded` audit event + `fail_permanent` (job → DEAD). On success, job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 0b — Key Data Extraction | None (regex) | Deterministic pre-processor: extracts numbers, percentages, currency amounts, ratios, formulas, underscore identifiers, and date ranges from source text via regex; appends them as a `[Key Data — extracted by pre-processor]` section before any LLM sees the text. Zero token cost; cannot hallucinate. Anchors the synthesis LLM to all quantitative candidates. |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads the full source text (bounded by `max_source_chars`) + BM25 candidates + `purpose.md` scope + status of each candidate page. Outputs per-page action: `create`, `update`, `flag`, `skip`. **RULE 1b:** a page with `status='active'` is human-reviewed and authoritative — when the source provides a conflicting value, date, formula, or conclusion, the action must be `flag`, not `update`. The only exception is a source that adds a topic the page does not yet mention at all. Decision cache version: `v2`. |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. If no citation markers are present in the returned body, a `citation_pass4_no_markers` WARNING audit event is written — the page is still saved but the condition is flagged for re-ingest with a more capable model. Zero-citation results are not cached so a subsequent re-ingest can succeed. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic. **Suppressed when TF fallback was used** (a small corpus with broad coverage may legitimately return fewer than 3 candidates — that is expected, not a gap).\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n| Citation presence (Check 5b) | Page body has ≥ 50 words and zero `^[filename:L-L]` citation markers → WARNING. Diagnostic only — does not block promotion to `active`. Indicates the annotation pass failed to produce markers, usually a model-compatibility issue (use Gemini 2.5 Flash or higher for reliable citation annotation). |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n### ActionAgent\n\nDispatches action-intent queries from the chat UI (e.g. \"activate a draft page\", \"show lint report\", \"archive a stale page\", \"run scaffold\"). Parses the user's free-text intent into a structured action via an LLM extraction prompt, then executes the action against the wiki and returns a human-readable result. When the intent is ambiguous (e.g. \"activate a page\" without specifying which), sets `needs_clarification=True` and returns a prompt and candidate list for the web UI to render as chip buttons.\n\n### ScaffoldAgent\n\nGenerates and updates the wiki's structural pages. Reads all current wiki pages and asks the LLM to organise them into 5–8 domain-appropriate categories, then writes `wiki/index.md` (category index with wikilinks), `wiki/purpose.md` (scope statement), and `AGENTS.md` (ingest and query guidelines). Also stamps a `categories:` field on each page's YAML frontmatter. When `ROUTING.md` already exists, regenerates it to stay in sync with the updated index structure.\n\n### RewriteAgent\n\nRewrites follow-up questions in multi-turn conversations into self-contained, standalone form before BM25 retrieval. Converts context-dependent phrases (\"What came after that?\", \"Tell me more about his early life\") into explicit questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the correct pages without relying on conversation context. Only invoked when conversation history is non-empty. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### SummarizeAgent\n\nCompresses the oldest conversation turns into a single `[Session summary: …]` assistant turn when a session exceeds `conversation_history_turns`. Prevents unbounded context growth across long sessions. Emits a `notice` SSE event the first time compression occurs so the user can see that earlier context was condensed. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### ContextAgent\n\nBuilds token-budgeted context packs for MCP tool consumers. See [Section 20 — Context Packs](#20-context-packs) for full detail.\n\n### SearchDecomposeAgent\n\nDecomposes a web search intent into 1–4 terse keyword search strings optimised for authoritative source retrieval. Uses a separate prompt from `QueryAgent`'s question decomposition — search decomposition asks \"what distinct search strings would find the best sources?\" (terse keyword phrases) rather than \"what distinct questions does this ask?\" (natural-language sub-questions). See [Web search fan-out](#web-search-fan-out) in the IngestAgent section.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n**Compound identifier expansion:** When a token contains underscores, the tokenizer emits both the compound form and each component as separate tokens (`capex_growth` → `[\"capex_growth\", \"capex\", \"growth\"]`). This expansion is applied at both index time and query time so that human-friendly queries match programmatic identifiers and vice versa. Leading/trailing underscores are stripped before splitting; no empty strings are emitted.\n\n**TF fallback for small corpora:** When all BM25 scores in a result set are zero or negative (IDF collapse on a corpus of ≤ 2 documents, or a term present in every page), the search system falls back to normalised term frequency (TF) scoring: `TF(doc, query) = Σcount(term, doc) / len(doc_tokens)`. Results are returned in the same format as BM25 results with a `tf_fallback: True` flag on each `SearchResult`. This field is recorded in the audit trail and consumed by gap detection (see below).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp, cascade_links_removed_from: [str]}` _(cascade field added v1.0.2)_ |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n| `GET` | `/graph` _(v1.0.0)_ | — | `{status, node_count, edge_count, cluster_count, nodes: [...], edges: [...]}` or `{status: \"computing\"}` on first call |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 1.0.2\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n**Reading View default:** `synthadoc plugin install` and `synthadoc plugin upgrade` write `\"defaultViewMode\": \"preview\"` to `.obsidian/app.json` after copying the plugin files. This causes Obsidian to open new notes in Reading View by default, which is required for `^[filename:L-L]` citation chips to render. Only `defaultViewMode` is written; all other keys in `app.json` are preserved. If `app.json` is malformed JSON the write is skipped and a warning is logged — the install still succeeds.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports two things: dangling slugs (pages listed in ROUTING.md that no longer exist in the wiki) and unassigned slugs (pages that exist in index.md but are not listed in any ROUTING.md branch). Enabled only when ROUTING.md exists. **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [-w wiki] — generate ROUTING.md from index.md branch structure\n│ ├── validate [-w wiki] — report dangling slugs and cross-branch duplicates\n│ └── clean [-w wiki] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [-w wiki]\n├── candidates\n│ ├── list [-w wiki]\n│ ├── promote |--all [-w wiki]\n│ └── discard |--all [-w wiki]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n# Citation pass (Pass 4) tuning — these two settings work together:\n# citation_source_lines — how many lines of the source the LLM sees when placing ^[...] markers.\n# Increase if lint reports out_of_range on long sources (transcripts, PDFs).\n# citation_max_tokens — output token budget for the annotated section returned by the LLM.\n# Increase if you raise citation_source_lines and have long wiki sections.\n# citation_source_lines = 400\n# citation_max_tokens = 8192\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Emit `logger.warning` in server log when per-job ingest cost exceeds this threshold; job continues normally |\n| `cost.hard_gate_usd` | float | `2.00` | Permanently fail the ingest job (DEAD) and record a `cost_gate_exceeded` audit event when per-job cost exceeds this threshold |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × (backup_count + 1)` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n| `ingest.max_source_chars` | int | `32000` | Character limit applied to each source before the LLM call. Sources exceeding this limit are truncated; the page's `sources:` frontmatter entry gets `truncated: true` and lint emits a warning. Override per-run with `--max-source-chars N`. _(v1.0.0)_ |\n| `ingest.citation_source_lines` | int | `400` | Number of source lines the LLM sees during Pass 4 (citation annotation). Increase if lint reports `out_of_range` on long sources such as transcripts or PDFs. Raise `citation_max_tokens` in proportion when increasing this value. |\n| `ingest.citation_max_tokens` | int | `8192` | Output token budget for the annotated section returned by Pass 4. Increase when `citation_source_lines` is raised or wiki sections are long, to avoid truncated annotations. |\n| `query.context_wiki_pct` | float | `0.60` | Fraction of the model context window reserved for wiki source pages. _(v1.0.0)_ |\n| `query.context_history_pct` | float | `0.20` | Fraction reserved for conversation history. _(v1.0.0)_ |\n| `query.context_system_pct` | float | `0.15` | Fraction reserved for system prompt and instructions. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_index_pct` | float | `0.05` | Fraction reserved for the wiki index preamble. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_window` | int | _(auto)_ | Override the built-in context window lookup for the configured model. Use when running a local or custom model whose window size is not in the built-in table. _(v1.0.0)_ |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement is wired into the ingest pipeline: `Orchestrator._guard_ingest_cost()` is called after each ingest LLM call with the returned cost, before the job is marked complete.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | `logger.warning` in server log; job continues normally |\n| `hard_gate_usd` | $2.00 | Server/ingest path: records `cost_gate_exceeded` audit event + permanently fails job (DEAD). Interactive CLI: prompts `Proceed? [y/N]` |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` by default. The bind address is configurable via `server.host` in `config.toml` (e.g. `host = \"0.0.0.0\"` to expose on all interfaces for LAN access). No built-in authentication — restrict via firewall when exposing beyond loopback. Remote access without a reverse proxy is not recommended on shared or networked machines.\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs and unassigned slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `-w ` / `--wiki ` to target a specific wiki.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; when all local source files are missing from disk the page is transitioned to `archived` (if only some sources are missing it is transitioned to `stale` instead). Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: lint evaluates all local (non-URL) source paths listed in the page's frontmatter. If **all** of them are missing from disk, the page is transitioned to `archived`. If only **some** are missing — meaning valid content from at least one source still exists — the page is transitioned to `stale` instead, preserving it for re-ingest rather than discarding it. _(v1.0.2: previously any single missing source triggered an immediate archive.)_\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Cascade link cleanup\n\nWhenever a page transitions to `archived` — whether triggered manually (`synthadoc lifecycle archive`), via the Obsidian Lifecycle modal, the MCP `synthadoc_lifecycle` tool, or automatically by lint detecting a missing source — Synthadoc immediately scans all other active pages and removes every `[[archived-slug]]` wikilink pointing to the now-archived page.\n\n- **Inline links** — `[[archived-slug]]` becomes plain text (the display text if an alias was used, otherwise the slug itself).\n- **List-item links** — a list item whose only content is `[[archived-slug]]` is dropped entirely.\n- **Archived pages are skipped** — pages already archived are not rewritten.\n- **System pages are skipped** — index, overview, dashboard, log, and purpose pages are excluded.\n\nThe cleanup runs synchronously before the response is returned, so the wiki is consistent immediately. The CLI and MCP responses include a `cascade_links_removed_from` field listing the affected slugs. During `lint run`, cascade is batched once after all auto-archive transitions complete, and the count is added to `dangling_links_removed` in the lint report.\n\nLint's existing periodic dangling-link cleanup (`lint run --scope orphans`) remains in place as a safety net for any dead links that predate this feature or arrive through other paths.\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 5) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 5 # turns to include in each request (default: 5; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp, cascade_links_removed_from: [str]}` or `{error, cascade_links_removed_from: []}` _(cascade field added v1.0.2)_ | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── hooks/ ← user hook scripts (always included)\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ ├── extracted/ ← text sidecars + PDF pagemaps (always included)\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `logs/`.\n\n> **v0.9.3 fix:** `hooks/` and `.synthadoc/extracted/` were previously missing from backups. Omitting `extracted/` caused the Source Viewer and Provenance citation modal to show \"Could not read source file\" after restore.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## 29. Pre-LLM Source Sanitizer\n\nEvery source document passes through a sanitizer immediately after text extraction, before any LLM call. The sanitizer removes six categories of content that could manipulate the LLM or degrade compilation quality:\n\n| Category | Action | Warning logged? |\n|---|---|---|\n| Zero-width characters (U+200B, U+200C, U+200D, U+FEFF) | Removed silently | No |\n| Bidi override characters (U+202A–U+202E, U+2066–U+2069) | Removed | Yes |\n| HTML comments (``) | Removed silently | No |\n| Hidden CSS spans (`display:none`, `visibility:hidden`) | Removed silently | No |\n| Base64 blobs ≥ 200 consecutive characters | Replaced with `[base64 content removed]` | Yes |\n| Instruction-override phrases (8 patterns: \"ignore previous instructions\", \"disregard the above\", \"override your system prompt\", etc.) | Replaced with `[redacted]` | Always |\n\nWhen content is removed and a warning is appropriate, the server logs a `WARN` entry:\n```\n[WARN] sanitizer stripped content from 'papers/survey.pdf': bidi overrides, instruction-override phrase\n```\n\nThis is a pure pre-processing step with zero LLM cost. Truncation (if configured) is applied after sanitization, so the character budget is consumed by clean text only.\n\n---\n\n## 30. Per-Source Truncation Flag\n\nLarge source documents are truncated before they reach the LLM. By default the limit is **32,000 characters** (~8,000 tokens), raised from the prior hardcoded 8,000-character limit. When a source exceeds this limit, its compiled wiki page records a `truncated: true` flag in its `sources:` frontmatter entry.\n\n### Configuration\n\n```toml\n[ingest]\nmax_source_chars = 32000 # default; raise for large PDFs or books\n```\n\n### Per-ingest override\n\nThe limit can be raised for a single ingest run without modifying the config:\n\n```bash\nsynthadoc ingest papers/large-textbook.pdf --max-source-chars 128000\n```\n\nThe same `max_source_chars` field is accepted by the HTTP `POST /jobs/ingest` body and the MCP `synthadoc_ingest` tool.\n\n### Lint surfacing\n\n`synthadoc lint` emits a warning for any page that has a truncated source:\n\n```\n[WARN] quantum-computing.md: source 'papers/large-survey.pdf' was truncated at ingest\n (source exceeded max_source_chars=32000 — 87,412 chars in source).\n To re-ingest with a higher limit (this source only):\n synthadoc ingest papers/large-survey.pdf --max-source-chars 128000\n To raise the limit for all future ingests:\n set [ingest] max_source_chars = 128000 in your config\n```\n\n---\n\n## 31. Proportional Context Budget\n\nQuery context is allocated proportionally to the configured model's context window, replacing the prior hard cap. The default allocation:\n\n| Slice | Default | Purpose |\n|---|---|---|\n| `context_wiki_pct` | 60% | Wiki page content (ranked by BM25 + vector score) |\n| `context_history_pct` | 20% | Chat turn history (newest-first, oldest dropped when over budget) |\n| `context_system_pct` | 15% | System prompt and purpose document |\n| `context_index_pct` | 5% | `ROUTING.md` and search index |\n\nPercentages must sum to ≤ 100. Configure in `[query]`:\n\n```toml\n[query]\ncontext_window = 0 # 0 = auto-detect from known model map\ncontext_wiki_pct = 60\ncontext_history_pct = 20\ncontext_system_pct = 15\ncontext_index_pct = 5\n```\n\n> **Note:** `context_system_pct` and `context_index_pct` are parsed and validated but not yet enforced as hard caps in this release — the system prompt uses a conservative fixed limit, which is well within the configured 15% slice for all supported models. Full per-slice enforcement is planned for v1.1.\n\n### Model context window map\n\nSynthadoc ships a built-in prefix-matched table:\n\n| Model | Context window |\n|---|---|\n| `claude-opus-4*`, `claude-sonnet-4*`, `claude-haiku-4*` | 200,000 tokens |\n| `gpt-4o*`, `gpt-4-turbo*` | 128,000 tokens |\n| `gpt-4` (exact) | 8,192 tokens |\n| `gpt-3.5-turbo*` | 16,385 tokens |\n| Unknown / fallback | 128,000 tokens |\n\nSet `context_window = N` in config to override the map (useful for local Ollama models with non-standard context sizes).\n\nAt the defaults on a 200k-token model: wiki slice ≈ 480,000 chars — far more than any practical query needs. On constrained models, the budget scales down automatically so the same config works across model families.\n\n---\n\n## 32. Knowledge Graph\n\nSynthadoc computes a wikilink graph across all active and draft pages during every lint run. The graph is stored in two tables in `audit.db`:\n\n```sql\nCREATE TABLE graph_nodes (\n slug TEXT PRIMARY KEY,\n cluster_id INTEGER NOT NULL,\n updated_at TEXT NOT NULL\n);\n\nCREATE TABLE graph_edges (\n from_slug TEXT NOT NULL,\n to_slug TEXT NOT NULL,\n weight INTEGER NOT NULL DEFAULT 1,\n PRIMARY KEY (from_slug, to_slug)\n);\n```\n\nEdge weight equals the number of `[[wikilink]]` references from one page to another (most pairs: 1). Cluster IDs are assigned by Louvain community detection — pages that link densely to each other end up in the same cluster.\n\n### REST API\n\n**`GET /graph`** — returns the full graph with per-node enrichment:\n\n```json\n{\n \"status\": \"ready\",\n \"node_count\": 45,\n \"edge_count\": 123,\n \"cluster_count\": 4,\n \"nodes\": [\n {\"slug\": \"quantum-error-correction\", \"title\": \"Quantum Error Correction\",\n \"type\": \"concept\", \"state\": \"active\", \"cluster_id\": 2}\n ],\n \"edges\": [\n {\"from\": \"quantum-error-correction\", \"to\": \"shor-algorithm\", \"weight\": 1}\n ]\n}\n```\n\nWhen the graph tables are empty (first call after upgrade, or before the first lint run), the server returns `{\"status\": \"computing\"}` and starts a background build immediately. The web UI polls every 2 seconds and renders the graph when the status becomes `\"ready\"`.\n\n### Web UI\n\nThe **Graph** tab (alongside Chat in the top nav) renders a D3.js force-directed graph. Nodes are colored by cluster. Clicking a node opens a sidebar showing the page title, type and lifecycle state badges, cluster assignment, and an **\"Ask about this →\"** button that switches to the Chat tab with a pre-filled query.\n\nControls: zoom in/out (scroll or pinch), drag nodes, filter by page type.\n\n---\n\n## Customization\n\nSynthadoc exposes four extension points — all hot-loaded, no server restart required.\n\n### Custom skills (new file formats)\n\nSubclass `BaseSkill` (Apache-2.0 — no AGPL obligation on your skill code), drop the folder in `/skills/` or `~/.synthadoc/skills/`, and Synthadoc picks it up on the next ingest. Skills match by file extension or intent prefix and support any Unicode text, including CJK prefixes.\n\n→ Full interface, manifest format, and examples: [§5 Skills System](#5-skills-system) and [§17 Plugin Development Guide](#17-plugin-development-guide)\n\n### Custom LLM providers\n\nSubclass `LLMProvider` from `synthadoc/providers/base.py` (Apache-2.0) and place the file in `~/.synthadoc/providers/` or the wiki `providers/` directory. Switch active provider with one config line.\n\n→ Full interface and wiring instructions: [§10 Configuration — Provider switching](#10-configuration)\n\n### Hooks\n\nShell scripts (any language) that fire on `on_ingest_complete` and `on_lint_complete`. Receive a JSON context on stdin. Set `blocking = true` to gate the operation on the hook's exit code — useful for CI pipelines and quality gates.\n\n→ Full event schema, context JSON, and blocking behaviour: [§11 Hook System](#11-hook-system)\n\n### Cache control\n\nThree cache layers (embedding, LLM response, provider prompt cache) invalidate automatically on source-file change (SHA-256). Force a fresh call with `--force`, or wipe all cached responses with `synthadoc cache clear -w `.\n\n→ Layer-by-layer breakdown and invalidation rules: [§12 Cache System](#12-cache-system)\n\n### Multi-Platform Agent Skill Files\n\nEvery Synthadoc wiki ships three companion files that give AI coding agents a complete, self-contained operating guide for the wiki:\n\n| File | Platform |\n|---|---|\n| `AGENTS.md` | Codex CLI, OpenCode, and any agent that reads the AGENTS.md convention |\n| `CLAUDE.md` | Claude Code (this session's tool) — highest-priority instruction source in Claude's hierarchy |\n| `GEMINI.md` | Gemini CLI — relevant because Synthadoc's default LLM provider is `gemini-2.5-flash-lite` |\n\nAll three files share identical body content generated from the same template; they differ only in their H1 heading so each agent recognises its own file first. The body covers:\n\n- **Domain guidelines** — LLM-generated bullet list of domain-specific ingest and query rules (produced by `scaffold`)\n- **Quick reference table** — every common CLI command with the correct `synthadoc` syntax\n- **Server startup** — how to start the server and verify it is running\n- **Ingest examples** — local file, URL, YouTube, agent session `.jsonl`, force reingest, dry-run\n- **Query** — basic and streaming query syntax; citation marker format\n- **Lint** — when to run and what each check does\n- **Lifecycle** — transition commands with state names\n- **Page schema** — frontmatter structure with all fields\n- **MCP tools table** — all 12 tools with purpose descriptions\n\n**How the files are created and updated:**\n\n1. `synthadoc init` — writes all three files with default domain guidelines and the wiki's configured port.\n2. `synthadoc scaffold` — regenerates all three with LLM-produced domain guidelines derived from the current wiki state. The `scaffold` command prints a confirmation line for each file.\n\n**Use cases:**\n\n- Open Claude Code in a Synthadoc wiki directory and it immediately knows how to ingest, query, lint, and manage lifecycle — no additional setup. Claude Code can also start the server in background mode (`synthadoc serve -w -b`), stop it cleanly, and orchestrate multi-step workflows, effectively acting as a control plane for the wiki.\n- Gemini CLI reads `GEMINI.md` and can run `synthadoc` commands with the correct flags without asking the user for help.\n- CI pipelines with Codex CLI use `AGENTS.md` to drive automated lint and lifecycle transitions on PR merge.\n- Agent session files produced by Claude Code (`.jsonl` under `~/.claude/projects/`) can themselves be ingested into the wiki, capturing the reasoning behind design decisions alongside the source documents.\n\n→ See [README.md — Interfaces & Integration](../README.md#interfaces--integration) for the feature comparison row and quick-start instructions.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v1.0.2\n\n- **Cascade link cleanup on archive** — archiving any page immediately removes every `[[slug]]` wikilink pointing to it from all other active pages, without waiting for the next `lint run`. Inline links become plain text; list-item-only links are dropped entirely. The HTTP `POST /lifecycle/transition` and MCP `synthadoc_lifecycle` responses now include a `cascade_links_removed_from` field listing the rewritten page slugs. During `lint run`, cascade runs once after all auto-archive transitions complete and the count is included in the lint report's `dangling_links_removed` field. System pages (index, dashboard, purpose, log, overview) and already-archived pages are skipped. The existing dangling-link lint pass remains as a safety net for dead links that predate this release.\n- **Multi-source partial-archive fix** — when a page has multiple local source files and only some are missing from disk, lint now transitions the page to `stale` (not `archived`), preserving it for re-ingest. The page is only archived when every local source file is gone. URL sources are not affected by this check.\n\n### v1.0.1\n\n- **Scaffold JSON extraction hardened** — replaced the greedy regex used to extract the LLM's JSON response with a brace-balanced parser; eliminates false-positive matches on `[[wikilink]]` syntax that previously caused `json.loads()` failures and silent scaffold corruption\n- **Generated file validation** — scaffold now validates every LLM-generated file (index, purpose, AGENTS.md) for minimum length, required frontmatter, and `[[wikilink]]` presence before writing to disk; validation failure marks the job as failed with a structured error message rather than silently writing a malformed page\n- **ROUTING.md regeneration validation** — `routing init` and scheduled routing regeneration now validate the regenerated file for minimum line count and anchor presence before applying it; prevents a corrupt or truncated ROUTING.md from entering the wiki\n- **Scaffold failure reporting** — scaffold skip and failure reasons are now surfaced in the terminal output and job error field; previously silent failures (e.g. LLM timeout, JSON parse error) are now visible without tailing the JSON log\n- **Install-time scaffold removed** — `synthadoc install` no longer runs the LLM scaffold call during installation; scaffold is a separate step (`synthadoc scaffold`) that the user runs after setting their API key, avoiding confusing API-key-not-set errors at install time\n- **AquaFlow LLM evaluation** — 15-question M&A due-diligence benchmark across five models (MiniMax-Think M3, Claude Opus 4.8, Claude Sonnet 4.6, DeepSeek-R1, Qwen Plus); full report at `docs/example/aquaflow/evaluation/report/llm-query-benchmark.md`\n\n### v1.0.0\n\n- **Pre-LLM source sanitizer** — every source document passes through a sanitizer before any LLM call; removes six categories of potentially harmful content (zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars, instruction-override phrases); a `WARN` log entry is emitted when bidi overrides or instruction-override phrases are stripped; zero LLM cost; applied before truncation so the character budget is consumed by clean text only\n- **Per-source truncation flag** — default character limit raised from 8,000 to 32,000 characters per source (~8,000 tokens); truncated sources are flagged with `truncated: true` in the page's `sources:` frontmatter; `--max-source-chars N` CLI flag and matching `POST /jobs/ingest` body field and MCP `synthadoc_ingest` parameter override the limit per run; `synthadoc lint` emits a warning for any page with a truncated source and suggests the override command; configurable via `[ingest] max_source_chars` in `config.toml`\n- **Proportional context budget** — query context is allocated proportionally to the configured model's context window; four configurable slices: `context_wiki_pct` (60%), `context_history_pct` (20%), `context_system_pct` (15%), `context_index_pct` (5%); built-in prefix-matched model context window table covers Claude 4, GPT-4o/turbo/3.5, and GPT-4; unknown models fall back to 128,000 tokens; `context_window = N` in `[query]` overrides the table for local or custom models\n- **Knowledge graph** — wikilink graph computed across all active and draft pages during every lint run; stored in `graph_nodes` and `graph_edges` tables in `audit.db`; edge weight = number of `[[wikilink]]` references between two pages; cluster IDs assigned by Louvain community detection; `GET /graph` REST endpoint returns nodes with per-node enrichment (title, type, state, cluster) and edges; first call after upgrade triggers a background build and returns `{\"status\":\"computing\"}`; web UI **Graph** tab (D3.js force-directed graph) — nodes colored by cluster, click to see page details and an \"Ask about this →\" button that opens a pre-filled chat query\n- **Citation accuracy & rendering improvements** — six correctness fixes to the ingest citation pipeline: (1) full source text passed to the decision LLM so citations can reference any line in the document; (2) `_annotate_citations` bugs A–F resolved (false-positive matches, off-by-one line ranges, duplicate markers, escaped bracket handling, multi-paragraph span errors, and source filename normalisation); (3) `_has_citations` check added — lint warns and suggests a model upgrade when a page has zero citation markers after synthesis; (4) `Key Data` section extraction hardened so numerical facts, rates, and formulas are preserved verbatim across the annotation pass; (5) `obsidianCitationsToGfm` rendering fixed in the web UI — citation superscripts now display correctly in the chat panel alongside streamed answers; (6) reading view set as the default Obsidian display mode for wiki pages\n\n### v0.9.3\n\n- **Backup now includes `hooks/` and `.synthadoc/extracted/`** — both directories were silently omitted from backups. `hooks/` contains user-customised hook scripts; `.synthadoc/extracted/` contains per-source text sidecars and PDF pagemaps written by the ingest agent. Without these, the Source Viewer and Provenance citation modal showed \"Could not read source file\" after restore.\n- **PyPI publish workflow** — `publish.yml` triggers on GitHub Release, builds the wheel with hatchling, and uploads to PyPI via OIDC trusted publishing (no API token stored). `pip install synthadoc` now works.\n- **Web UI bundled in pip package** — `synthadoc/data/web-ui/dist/` is now included in the wheel. `synthadoc web` previously returned 503 on pip-installed instances because the React dist was not packaged.\n- **CI sync checks** — PRs now fail if `synthadoc/data/obsidian-plugin/`, `synthadoc/data/web-ui/`, or `README-pypi.md` are out of sync with their source. All sync-check steps use `shell: bash` for Windows runner compatibility.\n- **`bump_version.py` extended** — now also updates `synthadoc/data/obsidian-plugin/manifest.json` alongside the source manifest, preventing version drift between the bundled copy and the plugin source.\n\n### v0.9.2\n\n- **`synthadoc/data/web-ui/dist/` packaged** — web UI dist files committed to the repo and bundled in the wheel via hatchling artifacts.\n- **`scripts/sync_web_ui.py`** — helper script to sync `web-ui/dist/` into `synthadoc/data/web-ui/dist/` after `npm run build`.\n\n### v0.9.1\n\n- **PyPI metadata** — added `readme`, `keywords`, `classifiers`, and `[project.urls]` to `pyproject.toml` for correct PyPI rendering.\n- **`README-pypi.md`** — generated from `README.md` by `scripts/prepare_pypi_readme.py`; strips `pypi-strip` blocks and rewrites relative links to absolute GitHub URLs.\n- **Branch protection compatibility** — removed post-merge `build-plugin` CI job that pushed compiled plugin directly to `main`. Plugin sync is now enforced in PRs: CI fails if `synthadoc/data/obsidian-plugin/` is out of sync.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 5). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n"}, "1c74c1bddb9b95e63aa9aa95d51ce364869aa38b": {"t": 1785767262.1601334, "n": 197720, "h": "5e514728b273635b", "c": "# Synthadoc — Design Document\n\n**Version:** 1.0.2 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n29. [Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer)\n30. [Per-Source Truncation Flag](#30-per-source-truncation-flag)\n31. [Proportional Context Budget](#31-proportional-context-budget)\n32. [Knowledge Graph](#32-knowledge-graph)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the full source text (bounded by `max_source_chars`) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `flag` / `skip`). Pages with `status='active'` are protected — sources that conflict with them trigger `flag` rather than `update` (RULE 1b).\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Orchestrator calls `_guard_ingest_cost()` on the returned cost: soft warn → `logger.warning` only; hard gate → `cost_gate_exceeded` audit event + `fail_permanent` (job → DEAD). On success, job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 0b — Key Data Extraction | None (regex) | Deterministic pre-processor: extracts numbers, percentages, currency amounts, ratios, formulas, underscore identifiers, and date ranges from source text via regex; appends them as a `[Key Data — extracted by pre-processor]` section before any LLM sees the text. Zero token cost; cannot hallucinate. Anchors the synthesis LLM to all quantitative candidates. |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads the full source text (bounded by `max_source_chars`) + BM25 candidates + `purpose.md` scope + status of each candidate page. Outputs per-page action: `create`, `update`, `flag`, `skip`. **RULE 1b:** a page with `status='active'` is human-reviewed and authoritative — when the source provides a conflicting value, date, formula, or conclusion, the action must be `flag`, not `update`. The only exception is a source that adds a topic the page does not yet mention at all. Decision cache version: `v2`. |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. If no citation markers are present in the returned body, a `citation_pass4_no_markers` WARNING audit event is written — the page is still saved but the condition is flagged for re-ingest with a more capable model. Zero-citation results are not cached so a subsequent re-ingest can succeed. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic. **Suppressed when TF fallback was used** (a small corpus with broad coverage may legitimately return fewer than 3 candidates — that is expected, not a gap).\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n| Citation presence (Check 5b) | Page body has ≥ 50 words and zero `^[filename:L-L]` citation markers → WARNING. Diagnostic only — does not block promotion to `active`. Indicates the annotation pass failed to produce markers, usually a model-compatibility issue (use Gemini 2.5 Flash or higher for reliable citation annotation). |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n### ActionAgent\n\nDispatches action-intent queries from the chat UI (e.g. \"activate a draft page\", \"show lint report\", \"archive a stale page\", \"run scaffold\"). Parses the user's free-text intent into a structured action via an LLM extraction prompt, then executes the action against the wiki and returns a human-readable result. When the intent is ambiguous (e.g. \"activate a page\" without specifying which), sets `needs_clarification=True` and returns a prompt and candidate list for the web UI to render as chip buttons.\n\n### ScaffoldAgent\n\nGenerates and updates the wiki's structural pages. Reads all current wiki pages and asks the LLM to organise them into 5–8 domain-appropriate categories, then writes `wiki/index.md` (category index with wikilinks), `wiki/purpose.md` (scope statement), and `AGENTS.md` (ingest and query guidelines). Also stamps a `categories:` field on each page's YAML frontmatter. When `ROUTING.md` already exists, regenerates it to stay in sync with the updated index structure.\n\n### RewriteAgent\n\nRewrites follow-up questions in multi-turn conversations into self-contained, standalone form before BM25 retrieval. Converts context-dependent phrases (\"What came after that?\", \"Tell me more about his early life\") into explicit questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the correct pages without relying on conversation context. Only invoked when conversation history is non-empty. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### SummarizeAgent\n\nCompresses the oldest conversation turns into a single `[Session summary: …]` assistant turn when a session exceeds `conversation_history_turns`. Prevents unbounded context growth across long sessions. Emits a `notice` SSE event the first time compression occurs so the user can see that earlier context was condensed. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### ContextAgent\n\nBuilds token-budgeted context packs for MCP tool consumers. See [Section 20 — Context Packs](#20-context-packs) for full detail.\n\n### SearchDecomposeAgent\n\nDecomposes a web search intent into 1–4 terse keyword search strings optimised for authoritative source retrieval. Uses a separate prompt from `QueryAgent`'s question decomposition — search decomposition asks \"what distinct search strings would find the best sources?\" (terse keyword phrases) rather than \"what distinct questions does this ask?\" (natural-language sub-questions). See [Web search fan-out](#web-search-fan-out) in the IngestAgent section.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n**Compound identifier expansion:** When a token contains underscores, the tokenizer emits both the compound form and each component as separate tokens (`capex_growth` → `[\"capex_growth\", \"capex\", \"growth\"]`). This expansion is applied at both index time and query time so that human-friendly queries match programmatic identifiers and vice versa. Leading/trailing underscores are stripped before splitting; no empty strings are emitted.\n\n**TF fallback for small corpora:** When all BM25 scores in a result set are zero or negative (IDF collapse on a corpus of ≤ 2 documents, or a term present in every page), the search system falls back to normalised term frequency (TF) scoring: `TF(doc, query) = Σcount(term, doc) / len(doc_tokens)`. Results are returned in the same format as BM25 results with a `tf_fallback: True` flag on each `SearchResult`. This field is recorded in the audit trail and consumed by gap detection (see below).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp, cascade_links_removed_from: [str]}` _(cascade field added v1.0.2)_ |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n| `GET` | `/graph` _(v1.0.0)_ | — | `{status, node_count, edge_count, cluster_count, nodes: [...], edges: [...]}` or `{status: \"computing\"}` on first call |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 1.0.2\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n**Reading View default:** `synthadoc plugin install` and `synthadoc plugin upgrade` write `\"defaultViewMode\": \"preview\"` to `.obsidian/app.json` after copying the plugin files. This causes Obsidian to open new notes in Reading View by default, which is required for `^[filename:L-L]` citation chips to render. Only `defaultViewMode` is written; all other keys in `app.json` are preserved. If `app.json` is malformed JSON the write is skipped and a warning is logged — the install still succeeds.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports two things: dangling slugs (pages listed in ROUTING.md that no longer exist in the wiki) and unassigned slugs (pages that exist in index.md but are not listed in any ROUTING.md branch). Enabled only when ROUTING.md exists. **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [-w wiki] — generate ROUTING.md from index.md branch structure\n│ ├── validate [-w wiki] — report dangling slugs and cross-branch duplicates\n│ └── clean [-w wiki] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [-w wiki]\n├── candidates\n│ ├── list [-w wiki]\n│ ├── promote |--all [-w wiki]\n│ └── discard |--all [-w wiki]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n# Citation pass (Pass 4) tuning — these two settings work together:\n# citation_source_lines — how many lines of the source the LLM sees when placing ^[...] markers.\n# Increase if lint reports out_of_range on long sources (transcripts, PDFs).\n# citation_max_tokens — output token budget for the annotated section returned by the LLM.\n# Increase if you raise citation_source_lines and have long wiki sections.\n# citation_source_lines = 400\n# citation_max_tokens = 8192\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Emit `logger.warning` in server log when per-job ingest cost exceeds this threshold; job continues normally |\n| `cost.hard_gate_usd` | float | `2.00` | Permanently fail the ingest job (DEAD) and record a `cost_gate_exceeded` audit event when per-job cost exceeds this threshold |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × (backup_count + 1)` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n| `ingest.max_source_chars` | int | `32000` | Character limit applied to each source before the LLM call. Sources exceeding this limit are truncated; the page's `sources:` frontmatter entry gets `truncated: true` and lint emits a warning. Override per-run with `--max-source-chars N`. _(v1.0.0)_ |\n| `ingest.citation_source_lines` | int | `400` | Number of source lines the LLM sees during Pass 4 (citation annotation). Increase if lint reports `out_of_range` on long sources such as transcripts or PDFs. Raise `citation_max_tokens` in proportion when increasing this value. |\n| `ingest.citation_max_tokens` | int | `8192` | Output token budget for the annotated section returned by Pass 4. Increase when `citation_source_lines` is raised or wiki sections are long, to avoid truncated annotations. |\n| `query.context_wiki_pct` | float | `0.60` | Fraction of the model context window reserved for wiki source pages. _(v1.0.0)_ |\n| `query.context_history_pct` | float | `0.20` | Fraction reserved for conversation history. _(v1.0.0)_ |\n| `query.context_system_pct` | float | `0.15` | Fraction reserved for system prompt and instructions. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_index_pct` | float | `0.05` | Fraction reserved for the wiki index preamble. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_window` | int | _(auto)_ | Override the built-in context window lookup for the configured model. Use when running a local or custom model whose window size is not in the built-in table. _(v1.0.0)_ |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement is wired into the ingest pipeline: `Orchestrator._guard_ingest_cost()` is called after each ingest LLM call with the returned cost, before the job is marked complete.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | `logger.warning` in server log; job continues normally |\n| `hard_gate_usd` | $2.00 | Server/ingest path: records `cost_gate_exceeded` audit event + permanently fails job (DEAD). Interactive CLI: prompts `Proceed? [y/N]` |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` by default. The bind address is configurable via `server.host` in `config.toml` (e.g. `host = \"0.0.0.0\"` to expose on all interfaces for LAN access). No built-in authentication — restrict via firewall when exposing beyond loopback. Remote access without a reverse proxy is not recommended on shared or networked machines.\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs and unassigned slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `-w ` / `--wiki ` to target a specific wiki.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; when all local source files are missing from disk the page is transitioned to `archived` (if only some sources are missing it is transitioned to `stale` instead). Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: lint evaluates all local (non-URL) source paths listed in the page's frontmatter. If **all** of them are missing from disk, the page is transitioned to `archived`. If only **some** are missing — meaning valid content from at least one source still exists — the page is transitioned to `stale` instead, preserving it for re-ingest rather than discarding it. _(v1.0.2: previously any single missing source triggered an immediate archive.)_\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Cascade link cleanup\n\nWhenever a page transitions to `archived` — whether triggered manually (`synthadoc lifecycle archive`), via the Obsidian Lifecycle modal, the MCP `synthadoc_lifecycle` tool, or automatically by lint detecting a missing source — Synthadoc immediately scans all other active pages and removes every `[[archived-slug]]` wikilink pointing to the now-archived page.\n\n- **Inline links** — `[[archived-slug]]` becomes plain text (the display text if an alias was used, otherwise the slug itself).\n- **List-item links** — a list item whose only content is `[[archived-slug]]` is dropped entirely.\n- **Archived pages are skipped** — pages already archived are not rewritten.\n- **System pages are skipped** — index, overview, dashboard, log, and purpose pages are excluded.\n\nThe cleanup runs synchronously before the response is returned, so the wiki is consistent immediately. The CLI and MCP responses include a `cascade_links_removed_from` field listing the affected slugs. During `lint run`, cascade is batched once after all auto-archive transitions complete, and the count is added to `dangling_links_removed` in the lint report.\n\nLint's existing periodic dangling-link cleanup (`lint run --scope orphans`) remains in place as a safety net for any dead links that predate this feature or arrive through other paths.\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 5) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 5 # turns to include in each request (default: 5; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp, cascade_links_removed_from: [str]}` or `{error, cascade_links_removed_from: []}` _(cascade field added v1.0.2)_ | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── hooks/ ← user hook scripts (always included)\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ ├── extracted/ ← text sidecars + PDF pagemaps (always included)\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `logs/`.\n\n> **v0.9.3 fix:** `hooks/` and `.synthadoc/extracted/` were previously missing from backups. Omitting `extracted/` caused the Source Viewer and Provenance citation modal to show \"Could not read source file\" after restore.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## 29. Pre-LLM Source Sanitizer\n\nEvery source document passes through a sanitizer immediately after text extraction, before any LLM call. The sanitizer removes six categories of content that could manipulate the LLM or degrade compilation quality:\n\n| Category | Action | Warning logged? |\n|---|---|---|\n| Zero-width characters (U+200B, U+200C, U+200D, U+FEFF) | Removed silently | No |\n| Bidi override characters (U+202A–U+202E, U+2066–U+2069) | Removed | Yes |\n| HTML comments (``) | Removed silently | No |\n| Hidden CSS spans (`display:none`, `visibility:hidden`) | Removed silently | No |\n| Base64 blobs ≥ 200 consecutive characters | Replaced with `[base64 content removed]` | Yes |\n| Instruction-override phrases (8 patterns: \"ignore previous instructions\", \"disregard the above\", \"override your system prompt\", etc.) | Replaced with `[redacted]` | Always |\n\nWhen content is removed and a warning is appropriate, the server logs a `WARN` entry:\n```\n[WARN] sanitizer stripped content from 'papers/survey.pdf': bidi overrides, instruction-override phrase\n```\n\nThis is a pure pre-processing step with zero LLM cost. Truncation (if configured) is applied after sanitization, so the character budget is consumed by clean text only.\n\n---\n\n## 30. Per-Source Truncation Flag\n\nLarge source documents are truncated before they reach the LLM. By default the limit is **32,000 characters** (~8,000 tokens), raised from the prior hardcoded 8,000-character limit. When a source exceeds this limit, its compiled wiki page records a `truncated: true` flag in its `sources:` frontmatter entry.\n\n### Configuration\n\n```toml\n[ingest]\nmax_source_chars = 32000 # default; raise for large PDFs or books\n```\n\n### Per-ingest override\n\nThe limit can be raised for a single ingest run without modifying the config:\n\n```bash\nsynthadoc ingest papers/large-textbook.pdf --max-source-chars 128000\n```\n\nThe same `max_source_chars` field is accepted by the HTTP `POST /jobs/ingest` body and the MCP `synthadoc_ingest` tool.\n\n### Lint surfacing\n\n`synthadoc lint` emits a warning for any page that has a truncated source:\n\n```\n[WARN] quantum-computing.md: source 'papers/large-survey.pdf' was truncated at ingest\n (source exceeded max_source_chars=32000 — 87,412 chars in source).\n To re-ingest with a higher limit (this source only):\n synthadoc ingest papers/large-survey.pdf --max-source-chars 128000\n To raise the limit for all future ingests:\n set [ingest] max_source_chars = 128000 in your config\n```\n\n---\n\n## 31. Proportional Context Budget\n\nQuery context is allocated proportionally to the configured model's context window, replacing the prior hard cap. The default allocation:\n\n| Slice | Default | Purpose |\n|---|---|---|\n| `context_wiki_pct` | 60% | Wiki page content (ranked by BM25 + vector score) |\n| `context_history_pct` | 20% | Chat turn history (newest-first, oldest dropped when over budget) |\n| `context_system_pct` | 15% | System prompt and purpose document |\n| `context_index_pct` | 5% | `ROUTING.md` and search index |\n\nPercentages must sum to ≤ 100. Configure in `[query]`:\n\n```toml\n[query]\ncontext_window = 0 # 0 = auto-detect from known model map\ncontext_wiki_pct = 60\ncontext_history_pct = 20\ncontext_system_pct = 15\ncontext_index_pct = 5\n```\n\n> **Note:** `context_system_pct` and `context_index_pct` are parsed and validated but not yet enforced as hard caps in this release — the system prompt uses a conservative fixed limit, which is well within the configured 15% slice for all supported models. Full per-slice enforcement is planned for v1.1.\n\n### Model context window map\n\nSynthadoc ships a built-in prefix-matched table:\n\n| Model | Context window |\n|---|---|\n| `claude-opus-4*`, `claude-sonnet-4*`, `claude-haiku-4*` | 200,000 tokens |\n| `gpt-4o*`, `gpt-4-turbo*` | 128,000 tokens |\n| `gpt-4` (exact) | 8,192 tokens |\n| `gpt-3.5-turbo*` | 16,385 tokens |\n| Unknown / fallback | 128,000 tokens |\n\nSet `context_window = N` in config to override the map (useful for local Ollama models with non-standard context sizes).\n\nAt the defaults on a 200k-token model: wiki slice ≈ 480,000 chars — far more than any practical query needs. On constrained models, the budget scales down automatically so the same config works across model families.\n\n---\n\n## 32. Knowledge Graph\n\nSynthadoc computes a wikilink graph across all active and draft pages during every lint run. The graph is stored in two tables in `audit.db`:\n\n```sql\nCREATE TABLE graph_nodes (\n slug TEXT PRIMARY KEY,\n cluster_id INTEGER NOT NULL,\n updated_at TEXT NOT NULL\n);\n\nCREATE TABLE graph_edges (\n from_slug TEXT NOT NULL,\n to_slug TEXT NOT NULL,\n weight INTEGER NOT NULL DEFAULT 1,\n PRIMARY KEY (from_slug, to_slug)\n);\n```\n\nEdge weight equals the number of `[[wikilink]]` references from one page to another (most pairs: 1). Cluster IDs are assigned by Louvain community detection — pages that link densely to each other end up in the same cluster.\n\n### REST API\n\n**`GET /graph`** — returns the full graph with per-node enrichment:\n\n```json\n{\n \"status\": \"ready\",\n \"node_count\": 45,\n \"edge_count\": 123,\n \"cluster_count\": 4,\n \"nodes\": [\n {\"slug\": \"quantum-error-correction\", \"title\": \"Quantum Error Correction\",\n \"type\": \"concept\", \"state\": \"active\", \"cluster_id\": 2}\n ],\n \"edges\": [\n {\"from\": \"quantum-error-correction\", \"to\": \"shor-algorithm\", \"weight\": 1}\n ]\n}\n```\n\nWhen the graph tables are empty (first call after upgrade, or before the first lint run), the server returns `{\"status\": \"computing\"}` and starts a background build immediately. The web UI polls every 2 seconds and renders the graph when the status becomes `\"ready\"`.\n\n### Web UI\n\nThe **Graph** tab (alongside Chat in the top nav) renders a D3.js force-directed graph. Nodes are colored by cluster. Clicking a node opens a sidebar showing the page title, type and lifecycle state badges, cluster assignment, and an **\"Ask about this →\"** button that switches to the Chat tab with a pre-filled query.\n\nControls: zoom in/out (scroll or pinch), drag nodes, filter by page type.\n\n---\n\n## Customization\n\nSynthadoc exposes four extension points — all hot-loaded, no server restart required.\n\n### Custom skills (new file formats)\n\nSubclass `BaseSkill` (Apache-2.0 — no AGPL obligation on your skill code), drop the folder in `/skills/` or `~/.synthadoc/skills/`, and Synthadoc picks it up on the next ingest. Skills match by file extension or intent prefix and support any Unicode text, including CJK prefixes.\n\n→ Full interface, manifest format, and examples: [§5 Skills System](#5-skills-system) and [§17 Plugin Development Guide](#17-plugin-development-guide)\n\n### Custom LLM providers\n\nSubclass `LLMProvider` from `synthadoc/providers/base.py` (Apache-2.0) and place the file in `~/.synthadoc/providers/` or the wiki `providers/` directory. Switch active provider with one config line.\n\n→ Full interface and wiring instructions: [§10 Configuration — Provider switching](#10-configuration)\n\n### Hooks\n\nShell scripts (any language) that fire on `on_ingest_complete` and `on_lint_complete`. Receive a JSON context on stdin. Set `blocking = true` to gate the operation on the hook's exit code — useful for CI pipelines and quality gates.\n\n→ Full event schema, context JSON, and blocking behaviour: [§11 Hook System](#11-hook-system)\n\n### Cache control\n\nThree cache layers (embedding, LLM response, provider prompt cache) invalidate automatically on source-file change (SHA-256). Force a fresh call with `--force`, or wipe all cached responses with `synthadoc cache clear -w `.\n\n→ Layer-by-layer breakdown and invalidation rules: [§12 Cache System](#12-cache-system)\n\n### Per-wiki AGENTS.md\n\nEdit `/AGENTS.md` to give the LLM domain-specific instructions — terminology, page-naming conventions, what to cross-reference. This is the highest-priority instruction source for every agent run against this wiki; the `scaffold` command regenerates a starter version from current wiki state.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v1.0.2\n\n- **Cascade link cleanup on archive** — archiving any page immediately removes every `[[slug]]` wikilink pointing to it from all other active pages, without waiting for the next `lint run`. Inline links become plain text; list-item-only links are dropped entirely. The HTTP `POST /lifecycle/transition` and MCP `synthadoc_lifecycle` responses now include a `cascade_links_removed_from` field listing the rewritten page slugs. During `lint run`, cascade runs once after all auto-archive transitions complete and the count is included in the lint report's `dangling_links_removed` field. System pages (index, dashboard, purpose, log, overview) and already-archived pages are skipped. The existing dangling-link lint pass remains as a safety net for dead links that predate this release.\n- **Multi-source partial-archive fix** — when a page has multiple local source files and only some are missing from disk, lint now transitions the page to `stale` (not `archived`), preserving it for re-ingest. The page is only archived when every local source file is gone. URL sources are not affected by this check.\n\n### v1.0.1\n\n- **Scaffold JSON extraction hardened** — replaced the greedy regex used to extract the LLM's JSON response with a brace-balanced parser; eliminates false-positive matches on `[[wikilink]]` syntax that previously caused `json.loads()` failures and silent scaffold corruption\n- **Generated file validation** — scaffold now validates every LLM-generated file (index, purpose, AGENTS.md) for minimum length, required frontmatter, and `[[wikilink]]` presence before writing to disk; validation failure marks the job as failed with a structured error message rather than silently writing a malformed page\n- **ROUTING.md regeneration validation** — `routing init` and scheduled routing regeneration now validate the regenerated file for minimum line count and anchor presence before applying it; prevents a corrupt or truncated ROUTING.md from entering the wiki\n- **Scaffold failure reporting** — scaffold skip and failure reasons are now surfaced in the terminal output and job error field; previously silent failures (e.g. LLM timeout, JSON parse error) are now visible without tailing the JSON log\n- **Install-time scaffold removed** — `synthadoc install` no longer runs the LLM scaffold call during installation; scaffold is a separate step (`synthadoc scaffold`) that the user runs after setting their API key, avoiding confusing API-key-not-set errors at install time\n- **AquaFlow LLM evaluation** — 15-question M&A due-diligence benchmark across five models (MiniMax-Think M3, Claude Opus 4.8, Claude Sonnet 4.6, DeepSeek-R1, Qwen Plus); full report at `docs/example/aquaflow/evaluation/report/llm-query-benchmark.md`\n\n### v1.0.0\n\n- **Pre-LLM source sanitizer** — every source document passes through a sanitizer before any LLM call; removes six categories of potentially harmful content (zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars, instruction-override phrases); a `WARN` log entry is emitted when bidi overrides or instruction-override phrases are stripped; zero LLM cost; applied before truncation so the character budget is consumed by clean text only\n- **Per-source truncation flag** — default character limit raised from 8,000 to 32,000 characters per source (~8,000 tokens); truncated sources are flagged with `truncated: true` in the page's `sources:` frontmatter; `--max-source-chars N` CLI flag and matching `POST /jobs/ingest` body field and MCP `synthadoc_ingest` parameter override the limit per run; `synthadoc lint` emits a warning for any page with a truncated source and suggests the override command; configurable via `[ingest] max_source_chars` in `config.toml`\n- **Proportional context budget** — query context is allocated proportionally to the configured model's context window; four configurable slices: `context_wiki_pct` (60%), `context_history_pct` (20%), `context_system_pct` (15%), `context_index_pct` (5%); built-in prefix-matched model context window table covers Claude 4, GPT-4o/turbo/3.5, and GPT-4; unknown models fall back to 128,000 tokens; `context_window = N` in `[query]` overrides the table for local or custom models\n- **Knowledge graph** — wikilink graph computed across all active and draft pages during every lint run; stored in `graph_nodes` and `graph_edges` tables in `audit.db`; edge weight = number of `[[wikilink]]` references between two pages; cluster IDs assigned by Louvain community detection; `GET /graph` REST endpoint returns nodes with per-node enrichment (title, type, state, cluster) and edges; first call after upgrade triggers a background build and returns `{\"status\":\"computing\"}`; web UI **Graph** tab (D3.js force-directed graph) — nodes colored by cluster, click to see page details and an \"Ask about this →\" button that opens a pre-filled chat query\n- **Citation accuracy & rendering improvements** — six correctness fixes to the ingest citation pipeline: (1) full source text passed to the decision LLM so citations can reference any line in the document; (2) `_annotate_citations` bugs A–F resolved (false-positive matches, off-by-one line ranges, duplicate markers, escaped bracket handling, multi-paragraph span errors, and source filename normalisation); (3) `_has_citations` check added — lint warns and suggests a model upgrade when a page has zero citation markers after synthesis; (4) `Key Data` section extraction hardened so numerical facts, rates, and formulas are preserved verbatim across the annotation pass; (5) `obsidianCitationsToGfm` rendering fixed in the web UI — citation superscripts now display correctly in the chat panel alongside streamed answers; (6) reading view set as the default Obsidian display mode for wiki pages\n\n### v0.9.3\n\n- **Backup now includes `hooks/` and `.synthadoc/extracted/`** — both directories were silently omitted from backups. `hooks/` contains user-customised hook scripts; `.synthadoc/extracted/` contains per-source text sidecars and PDF pagemaps written by the ingest agent. Without these, the Source Viewer and Provenance citation modal showed \"Could not read source file\" after restore.\n- **PyPI publish workflow** — `publish.yml` triggers on GitHub Release, builds the wheel with hatchling, and uploads to PyPI via OIDC trusted publishing (no API token stored). `pip install synthadoc` now works.\n- **Web UI bundled in pip package** — `synthadoc/data/web-ui/dist/` is now included in the wheel. `synthadoc web` previously returned 503 on pip-installed instances because the React dist was not packaged.\n- **CI sync checks** — PRs now fail if `synthadoc/data/obsidian-plugin/`, `synthadoc/data/web-ui/`, or `README-pypi.md` are out of sync with their source. All sync-check steps use `shell: bash` for Windows runner compatibility.\n- **`bump_version.py` extended** — now also updates `synthadoc/data/obsidian-plugin/manifest.json` alongside the source manifest, preventing version drift between the bundled copy and the plugin source.\n\n### v0.9.2\n\n- **`synthadoc/data/web-ui/dist/` packaged** — web UI dist files committed to the repo and bundled in the wheel via hatchling artifacts.\n- **`scripts/sync_web_ui.py`** — helper script to sync `web-ui/dist/` into `synthadoc/data/web-ui/dist/` after `npm run build`.\n\n### v0.9.1\n\n- **PyPI metadata** — added `readme`, `keywords`, `classifiers`, and `[project.urls]` to `pyproject.toml` for correct PyPI rendering.\n- **`README-pypi.md`** — generated from `README.md` by `scripts/prepare_pypi_readme.py`; strips `pypi-strip` blocks and rewrites relative links to absolute GitHub URLs.\n- **Branch protection compatibility** — removed post-merge `build-plugin` CI job that pushed compiled plugin directly to `main`. Plugin sync is now enforced in PRs: CI fails if `synthadoc/data/obsidian-plugin/` is out of sync.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 5). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n"}, "3f9b6a5183ce02b80dccaa20761b9c274176cce8": {"t": 1785767266.9972107, "n": 194651, "h": "b542006f26383257", "c": "# Synthadoc — Design Document\n\n**Version:** 1.0.1 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n29. [Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer)\n30. [Per-Source Truncation Flag](#30-per-source-truncation-flag)\n31. [Proportional Context Budget](#31-proportional-context-budget)\n32. [Knowledge Graph](#32-knowledge-graph)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the full source text (bounded by `max_source_chars`) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `flag` / `skip`). Pages with `status='active'` are protected — sources that conflict with them trigger `flag` rather than `update` (RULE 1b).\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Orchestrator calls `_guard_ingest_cost()` on the returned cost: soft warn → `logger.warning` only; hard gate → `cost_gate_exceeded` audit event + `fail_permanent` (job → DEAD). On success, job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 0b — Key Data Extraction | None (regex) | Deterministic pre-processor: extracts numbers, percentages, currency amounts, ratios, formulas, underscore identifiers, and date ranges from source text via regex; appends them as a `[Key Data — extracted by pre-processor]` section before any LLM sees the text. Zero token cost; cannot hallucinate. Anchors the synthesis LLM to all quantitative candidates. |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads the full source text (bounded by `max_source_chars`) + BM25 candidates + `purpose.md` scope + status of each candidate page. Outputs per-page action: `create`, `update`, `flag`, `skip`. **RULE 1b:** a page with `status='active'` is human-reviewed and authoritative — when the source provides a conflicting value, date, formula, or conclusion, the action must be `flag`, not `update`. The only exception is a source that adds a topic the page does not yet mention at all. Decision cache version: `v2`. |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. If no citation markers are present in the returned body, a `citation_pass4_no_markers` WARNING audit event is written — the page is still saved but the condition is flagged for re-ingest with a more capable model. Zero-citation results are not cached so a subsequent re-ingest can succeed. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic. **Suppressed when TF fallback was used** (a small corpus with broad coverage may legitimately return fewer than 3 candidates — that is expected, not a gap).\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n| Citation presence (Check 5b) | Page body has ≥ 50 words and zero `^[filename:L-L]` citation markers → WARNING. Diagnostic only — does not block promotion to `active`. Indicates the annotation pass failed to produce markers, usually a model-compatibility issue (use Gemini 2.5 Flash or higher for reliable citation annotation). |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n### ActionAgent\n\nDispatches action-intent queries from the chat UI (e.g. \"activate a draft page\", \"show lint report\", \"archive a stale page\", \"run scaffold\"). Parses the user's free-text intent into a structured action via an LLM extraction prompt, then executes the action against the wiki and returns a human-readable result. When the intent is ambiguous (e.g. \"activate a page\" without specifying which), sets `needs_clarification=True` and returns a prompt and candidate list for the web UI to render as chip buttons.\n\n### ScaffoldAgent\n\nGenerates and updates the wiki's structural pages. Reads all current wiki pages and asks the LLM to organise them into 5–8 domain-appropriate categories, then writes `wiki/index.md` (category index with wikilinks), `wiki/purpose.md` (scope statement), and `AGENTS.md` (ingest and query guidelines). Also stamps a `categories:` field on each page's YAML frontmatter. When `ROUTING.md` already exists, regenerates it to stay in sync with the updated index structure.\n\n### RewriteAgent\n\nRewrites follow-up questions in multi-turn conversations into self-contained, standalone form before BM25 retrieval. Converts context-dependent phrases (\"What came after that?\", \"Tell me more about his early life\") into explicit questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the correct pages without relying on conversation context. Only invoked when conversation history is non-empty. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### SummarizeAgent\n\nCompresses the oldest conversation turns into a single `[Session summary: …]` assistant turn when a session exceeds `conversation_history_turns`. Prevents unbounded context growth across long sessions. Emits a `notice` SSE event the first time compression occurs so the user can see that earlier context was condensed. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### ContextAgent\n\nBuilds token-budgeted context packs for MCP tool consumers. See [Section 20 — Context Packs](#20-context-packs) for full detail.\n\n### SearchDecomposeAgent\n\nDecomposes a web search intent into 1–4 terse keyword search strings optimised for authoritative source retrieval. Uses a separate prompt from `QueryAgent`'s question decomposition — search decomposition asks \"what distinct search strings would find the best sources?\" (terse keyword phrases) rather than \"what distinct questions does this ask?\" (natural-language sub-questions). See [Web search fan-out](#web-search-fan-out) in the IngestAgent section.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n**Compound identifier expansion:** When a token contains underscores, the tokenizer emits both the compound form and each component as separate tokens (`capex_growth` → `[\"capex_growth\", \"capex\", \"growth\"]`). This expansion is applied at both index time and query time so that human-friendly queries match programmatic identifiers and vice versa. Leading/trailing underscores are stripped before splitting; no empty strings are emitted.\n\n**TF fallback for small corpora:** When all BM25 scores in a result set are zero or negative (IDF collapse on a corpus of ≤ 2 documents, or a term present in every page), the search system falls back to normalised term frequency (TF) scoring: `TF(doc, query) = Σcount(term, doc) / len(doc_tokens)`. Results are returned in the same format as BM25 results with a `tf_fallback: True` flag on each `SearchResult`. This field is recorded in the audit trail and consumed by gap detection (see below).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp}` |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n| `GET` | `/graph` _(v1.0.0)_ | — | `{status, node_count, edge_count, cluster_count, nodes: [...], edges: [...]}` or `{status: \"computing\"}` on first call |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 1.0.1\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n**Reading View default:** `synthadoc plugin install` and `synthadoc plugin upgrade` write `\"defaultViewMode\": \"preview\"` to `.obsidian/app.json` after copying the plugin files. This causes Obsidian to open new notes in Reading View by default, which is required for `^[filename:L-L]` citation chips to render. Only `defaultViewMode` is written; all other keys in `app.json` are preserved. If `app.json` is malformed JSON the write is skipped and a warning is logged — the install still succeeds.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports two things: dangling slugs (pages listed in ROUTING.md that no longer exist in the wiki) and unassigned slugs (pages that exist in index.md but are not listed in any ROUTING.md branch). Enabled only when ROUTING.md exists. **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [-w wiki] — generate ROUTING.md from index.md branch structure\n│ ├── validate [-w wiki] — report dangling slugs and cross-branch duplicates\n│ └── clean [-w wiki] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [-w wiki]\n├── candidates\n│ ├── list [-w wiki]\n│ ├── promote |--all [-w wiki]\n│ └── discard |--all [-w wiki]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n# Citation pass (Pass 4) tuning — these two settings work together:\n# citation_source_lines — how many lines of the source the LLM sees when placing ^[...] markers.\n# Increase if lint reports out_of_range on long sources (transcripts, PDFs).\n# citation_max_tokens — output token budget for the annotated section returned by the LLM.\n# Increase if you raise citation_source_lines and have long wiki sections.\n# citation_source_lines = 400\n# citation_max_tokens = 8192\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Emit `logger.warning` in server log when per-job ingest cost exceeds this threshold; job continues normally |\n| `cost.hard_gate_usd` | float | `2.00` | Permanently fail the ingest job (DEAD) and record a `cost_gate_exceeded` audit event when per-job cost exceeds this threshold |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × (backup_count + 1)` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n| `ingest.max_source_chars` | int | `32000` | Character limit applied to each source before the LLM call. Sources exceeding this limit are truncated; the page's `sources:` frontmatter entry gets `truncated: true` and lint emits a warning. Override per-run with `--max-source-chars N`. _(v1.0.0)_ |\n| `ingest.citation_source_lines` | int | `400` | Number of source lines the LLM sees during Pass 4 (citation annotation). Increase if lint reports `out_of_range` on long sources such as transcripts or PDFs. Raise `citation_max_tokens` in proportion when increasing this value. |\n| `ingest.citation_max_tokens` | int | `8192` | Output token budget for the annotated section returned by Pass 4. Increase when `citation_source_lines` is raised or wiki sections are long, to avoid truncated annotations. |\n| `query.context_wiki_pct` | float | `0.60` | Fraction of the model context window reserved for wiki source pages. _(v1.0.0)_ |\n| `query.context_history_pct` | float | `0.20` | Fraction reserved for conversation history. _(v1.0.0)_ |\n| `query.context_system_pct` | float | `0.15` | Fraction reserved for system prompt and instructions. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_index_pct` | float | `0.05` | Fraction reserved for the wiki index preamble. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_window` | int | _(auto)_ | Override the built-in context window lookup for the configured model. Use when running a local or custom model whose window size is not in the built-in table. _(v1.0.0)_ |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement is wired into the ingest pipeline: `Orchestrator._guard_ingest_cost()` is called after each ingest LLM call with the returned cost, before the job is marked complete.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | `logger.warning` in server log; job continues normally |\n| `hard_gate_usd` | $2.00 | Server/ingest path: records `cost_gate_exceeded` audit event + permanently fails job (DEAD). Interactive CLI: prompts `Proceed? [y/N]` |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` by default. The bind address is configurable via `server.host` in `config.toml` (e.g. `host = \"0.0.0.0\"` to expose on all interfaces for LAN access). No built-in authentication — restrict via firewall when exposing beyond loopback. Remote access without a reverse proxy is not recommended on shared or networked machines.\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs and unassigned slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `-w ` / `--wiki ` to target a specific wiki.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; a missing source file triggers `archived`. Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: if the source path recorded in frontmatter no longer exists on disk, the page is transitioned to `archived`.\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 5) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 5 # turns to include in each request (default: 5; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp}` or `{error}` | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── hooks/ ← user hook scripts (always included)\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ ├── extracted/ ← text sidecars + PDF pagemaps (always included)\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `logs/`.\n\n> **v0.9.3 fix:** `hooks/` and `.synthadoc/extracted/` were previously missing from backups. Omitting `extracted/` caused the Source Viewer and Provenance citation modal to show \"Could not read source file\" after restore.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## 29. Pre-LLM Source Sanitizer\n\nEvery source document passes through a sanitizer immediately after text extraction, before any LLM call. The sanitizer removes six categories of content that could manipulate the LLM or degrade compilation quality:\n\n| Category | Action | Warning logged? |\n|---|---|---|\n| Zero-width characters (U+200B, U+200C, U+200D, U+FEFF) | Removed silently | No |\n| Bidi override characters (U+202A–U+202E, U+2066–U+2069) | Removed | Yes |\n| HTML comments (``) | Removed silently | No |\n| Hidden CSS spans (`display:none`, `visibility:hidden`) | Removed silently | No |\n| Base64 blobs ≥ 200 consecutive characters | Replaced with `[base64 content removed]` | Yes |\n| Instruction-override phrases (8 patterns: \"ignore previous instructions\", \"disregard the above\", \"override your system prompt\", etc.) | Replaced with `[redacted]` | Always |\n\nWhen content is removed and a warning is appropriate, the server logs a `WARN` entry:\n```\n[WARN] sanitizer stripped content from 'papers/survey.pdf': bidi overrides, instruction-override phrase\n```\n\nThis is a pure pre-processing step with zero LLM cost. Truncation (if configured) is applied after sanitization, so the character budget is consumed by clean text only.\n\n---\n\n## 30. Per-Source Truncation Flag\n\nLarge source documents are truncated before they reach the LLM. By default the limit is **32,000 characters** (~8,000 tokens), raised from the prior hardcoded 8,000-character limit. When a source exceeds this limit, its compiled wiki page records a `truncated: true` flag in its `sources:` frontmatter entry.\n\n### Configuration\n\n```toml\n[ingest]\nmax_source_chars = 32000 # default; raise for large PDFs or books\n```\n\n### Per-ingest override\n\nThe limit can be raised for a single ingest run without modifying the config:\n\n```bash\nsynthadoc ingest papers/large-textbook.pdf --max-source-chars 128000\n```\n\nThe same `max_source_chars` field is accepted by the HTTP `POST /jobs/ingest` body and the MCP `synthadoc_ingest` tool.\n\n### Lint surfacing\n\n`synthadoc lint` emits a warning for any page that has a truncated source:\n\n```\n[WARN] quantum-computing.md: source 'papers/large-survey.pdf' was truncated at ingest\n (source exceeded max_source_chars=32000 — 87,412 chars in source).\n To re-ingest with a higher limit (this source only):\n synthadoc ingest papers/large-survey.pdf --max-source-chars 128000\n To raise the limit for all future ingests:\n set [ingest] max_source_chars = 128000 in your config\n```\n\n---\n\n## 31. Proportional Context Budget\n\nQuery context is allocated proportionally to the configured model's context window, replacing the prior hard cap. The default allocation:\n\n| Slice | Default | Purpose |\n|---|---|---|\n| `context_wiki_pct` | 60% | Wiki page content (ranked by BM25 + vector score) |\n| `context_history_pct` | 20% | Chat turn history (newest-first, oldest dropped when over budget) |\n| `context_system_pct` | 15% | System prompt and purpose document |\n| `context_index_pct` | 5% | `ROUTING.md` and search index |\n\nPercentages must sum to ≤ 100. Configure in `[query]`:\n\n```toml\n[query]\ncontext_window = 0 # 0 = auto-detect from known model map\ncontext_wiki_pct = 60\ncontext_history_pct = 20\ncontext_system_pct = 15\ncontext_index_pct = 5\n```\n\n> **Note:** `context_system_pct` and `context_index_pct` are parsed and validated but not yet enforced as hard caps in this release — the system prompt uses a conservative fixed limit, which is well within the configured 15% slice for all supported models. Full per-slice enforcement is planned for v1.1.\n\n### Model context window map\n\nSynthadoc ships a built-in prefix-matched table:\n\n| Model | Context window |\n|---|---|\n| `claude-opus-4*`, `claude-sonnet-4*`, `claude-haiku-4*` | 200,000 tokens |\n| `gpt-4o*`, `gpt-4-turbo*` | 128,000 tokens |\n| `gpt-4` (exact) | 8,192 tokens |\n| `gpt-3.5-turbo*` | 16,385 tokens |\n| Unknown / fallback | 128,000 tokens |\n\nSet `context_window = N` in config to override the map (useful for local Ollama models with non-standard context sizes).\n\nAt the defaults on a 200k-token model: wiki slice ≈ 480,000 chars — far more than any practical query needs. On constrained models, the budget scales down automatically so the same config works across model families.\n\n---\n\n## 32. Knowledge Graph\n\nSynthadoc computes a wikilink graph across all active and draft pages during every lint run. The graph is stored in two tables in `audit.db`:\n\n```sql\nCREATE TABLE graph_nodes (\n slug TEXT PRIMARY KEY,\n cluster_id INTEGER NOT NULL,\n updated_at TEXT NOT NULL\n);\n\nCREATE TABLE graph_edges (\n from_slug TEXT NOT NULL,\n to_slug TEXT NOT NULL,\n weight INTEGER NOT NULL DEFAULT 1,\n PRIMARY KEY (from_slug, to_slug)\n);\n```\n\nEdge weight equals the number of `[[wikilink]]` references from one page to another (most pairs: 1). Cluster IDs are assigned by Louvain community detection — pages that link densely to each other end up in the same cluster.\n\n### REST API\n\n**`GET /graph`** — returns the full graph with per-node enrichment:\n\n```json\n{\n \"status\": \"ready\",\n \"node_count\": 45,\n \"edge_count\": 123,\n \"cluster_count\": 4,\n \"nodes\": [\n {\"slug\": \"quantum-error-correction\", \"title\": \"Quantum Error Correction\",\n \"type\": \"concept\", \"state\": \"active\", \"cluster_id\": 2}\n ],\n \"edges\": [\n {\"from\": \"quantum-error-correction\", \"to\": \"shor-algorithm\", \"weight\": 1}\n ]\n}\n```\n\nWhen the graph tables are empty (first call after upgrade, or before the first lint run), the server returns `{\"status\": \"computing\"}` and starts a background build immediately. The web UI polls every 2 seconds and renders the graph when the status becomes `\"ready\"`.\n\n### Web UI\n\nThe **Graph** tab (alongside Chat in the top nav) renders a D3.js force-directed graph. Nodes are colored by cluster. Clicking a node opens a sidebar showing the page title, type and lifecycle state badges, cluster assignment, and an **\"Ask about this →\"** button that switches to the Chat tab with a pre-filled query.\n\nControls: zoom in/out (scroll or pinch), drag nodes, filter by page type.\n\n---\n\n## Customization\n\nSynthadoc exposes four extension points — all hot-loaded, no server restart required.\n\n### Custom skills (new file formats)\n\nSubclass `BaseSkill` (Apache-2.0 — no AGPL obligation on your skill code), drop the folder in `/skills/` or `~/.synthadoc/skills/`, and Synthadoc picks it up on the next ingest. Skills match by file extension or intent prefix and support any Unicode text, including CJK prefixes.\n\n→ Full interface, manifest format, and examples: [§5 Skills System](#5-skills-system) and [§17 Plugin Development Guide](#17-plugin-development-guide)\n\n### Custom LLM providers\n\nSubclass `LLMProvider` from `synthadoc/providers/base.py` (Apache-2.0) and place the file in `~/.synthadoc/providers/` or the wiki `providers/` directory. Switch active provider with one config line.\n\n→ Full interface and wiring instructions: [§10 Configuration — Provider switching](#10-configuration)\n\n### Hooks\n\nShell scripts (any language) that fire on `on_ingest_complete` and `on_lint_complete`. Receive a JSON context on stdin. Set `blocking = true` to gate the operation on the hook's exit code — useful for CI pipelines and quality gates.\n\n→ Full event schema, context JSON, and blocking behaviour: [§11 Hook System](#11-hook-system)\n\n### Cache control\n\nThree cache layers (embedding, LLM response, provider prompt cache) invalidate automatically on source-file change (SHA-256). Force a fresh call with `--force`, or wipe all cached responses with `synthadoc cache clear -w `.\n\n→ Layer-by-layer breakdown and invalidation rules: [§12 Cache System](#12-cache-system)\n\n### Per-wiki AGENTS.md\n\nEdit `/AGENTS.md` to give the LLM domain-specific instructions — terminology, page-naming conventions, what to cross-reference. This is the highest-priority instruction source for every agent run against this wiki; the `scaffold` command regenerates a starter version from current wiki state.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v1.0.1\n\n- **Scaffold JSON extraction hardened** — replaced the greedy regex used to extract the LLM's JSON response with a brace-balanced parser; eliminates false-positive matches on `[[wikilink]]` syntax that previously caused `json.loads()` failures and silent scaffold corruption\n- **Generated file validation** — scaffold now validates every LLM-generated file (index, purpose, AGENTS.md) for minimum length, required frontmatter, and `[[wikilink]]` presence before writing to disk; validation failure marks the job as failed with a structured error message rather than silently writing a malformed page\n- **ROUTING.md regeneration validation** — `routing init` and scheduled routing regeneration now validate the regenerated file for minimum line count and anchor presence before applying it; prevents a corrupt or truncated ROUTING.md from entering the wiki\n- **Scaffold failure reporting** — scaffold skip and failure reasons are now surfaced in the terminal output and job error field; previously silent failures (e.g. LLM timeout, JSON parse error) are now visible without tailing the JSON log\n- **Install-time scaffold removed** — `synthadoc install` no longer runs the LLM scaffold call during installation; scaffold is a separate step (`synthadoc scaffold`) that the user runs after setting their API key, avoiding confusing API-key-not-set errors at install time\n- **AquaFlow LLM evaluation** — 15-question M&A due-diligence benchmark across five models (MiniMax-Think M3, Claude Opus 4.8, Claude Sonnet 4.6, DeepSeek-R1, Qwen Plus); full report at `docs/example/aquaflow/evaluation/report/llm-query-benchmark.md`\n\n### v1.0.0\n\n- **Pre-LLM source sanitizer** — every source document passes through a sanitizer before any LLM call; removes six categories of potentially harmful content (zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars, instruction-override phrases); a `WARN` log entry is emitted when bidi overrides or instruction-override phrases are stripped; zero LLM cost; applied before truncation so the character budget is consumed by clean text only\n- **Per-source truncation flag** — default character limit raised from 8,000 to 32,000 characters per source (~8,000 tokens); truncated sources are flagged with `truncated: true` in the page's `sources:` frontmatter; `--max-source-chars N` CLI flag and matching `POST /jobs/ingest` body field and MCP `synthadoc_ingest` parameter override the limit per run; `synthadoc lint` emits a warning for any page with a truncated source and suggests the override command; configurable via `[ingest] max_source_chars` in `config.toml`\n- **Proportional context budget** — query context is allocated proportionally to the configured model's context window; four configurable slices: `context_wiki_pct` (60%), `context_history_pct` (20%), `context_system_pct` (15%), `context_index_pct` (5%); built-in prefix-matched model context window table covers Claude 4, GPT-4o/turbo/3.5, and GPT-4; unknown models fall back to 128,000 tokens; `context_window = N` in `[query]` overrides the table for local or custom models\n- **Knowledge graph** — wikilink graph computed across all active and draft pages during every lint run; stored in `graph_nodes` and `graph_edges` tables in `audit.db`; edge weight = number of `[[wikilink]]` references between two pages; cluster IDs assigned by Louvain community detection; `GET /graph` REST endpoint returns nodes with per-node enrichment (title, type, state, cluster) and edges; first call after upgrade triggers a background build and returns `{\"status\":\"computing\"}`; web UI **Graph** tab (D3.js force-directed graph) — nodes colored by cluster, click to see page details and an \"Ask about this →\" button that opens a pre-filled chat query\n- **Citation accuracy & rendering improvements** — six correctness fixes to the ingest citation pipeline: (1) full source text passed to the decision LLM so citations can reference any line in the document; (2) `_annotate_citations` bugs A–F resolved (false-positive matches, off-by-one line ranges, duplicate markers, escaped bracket handling, multi-paragraph span errors, and source filename normalisation); (3) `_has_citations` check added — lint warns and suggests a model upgrade when a page has zero citation markers after synthesis; (4) `Key Data` section extraction hardened so numerical facts, rates, and formulas are preserved verbatim across the annotation pass; (5) `obsidianCitationsToGfm` rendering fixed in the web UI — citation superscripts now display correctly in the chat panel alongside streamed answers; (6) reading view set as the default Obsidian display mode for wiki pages\n\n### v0.9.3\n\n- **Backup now includes `hooks/` and `.synthadoc/extracted/`** — both directories were silently omitted from backups. `hooks/` contains user-customised hook scripts; `.synthadoc/extracted/` contains per-source text sidecars and PDF pagemaps written by the ingest agent. Without these, the Source Viewer and Provenance citation modal showed \"Could not read source file\" after restore.\n- **PyPI publish workflow** — `publish.yml` triggers on GitHub Release, builds the wheel with hatchling, and uploads to PyPI via OIDC trusted publishing (no API token stored). `pip install synthadoc` now works.\n- **Web UI bundled in pip package** — `synthadoc/data/web-ui/dist/` is now included in the wheel. `synthadoc web` previously returned 503 on pip-installed instances because the React dist was not packaged.\n- **CI sync checks** — PRs now fail if `synthadoc/data/obsidian-plugin/`, `synthadoc/data/web-ui/`, or `README-pypi.md` are out of sync with their source. All sync-check steps use `shell: bash` for Windows runner compatibility.\n- **`bump_version.py` extended** — now also updates `synthadoc/data/obsidian-plugin/manifest.json` alongside the source manifest, preventing version drift between the bundled copy and the plugin source.\n\n### v0.9.2\n\n- **`synthadoc/data/web-ui/dist/` packaged** — web UI dist files committed to the repo and bundled in the wheel via hatchling artifacts.\n- **`scripts/sync_web_ui.py`** — helper script to sync `web-ui/dist/` into `synthadoc/data/web-ui/dist/` after `npm run build`.\n\n### v0.9.1\n\n- **PyPI metadata** — added `readme`, `keywords`, `classifiers`, and `[project.urls]` to `pyproject.toml` for correct PyPI rendering.\n- **`README-pypi.md`** — generated from `README.md` by `scripts/prepare_pypi_readme.py`; strips `pypi-strip` blocks and rewrites relative links to absolute GitHub URLs.\n- **Branch protection compatibility** — removed post-merge `build-plugin` CI job that pushed compiled plugin directly to `main`. Plugin sync is now enforced in PRs: CI fails if `synthadoc/data/obsidian-plugin/` is out of sync.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 5). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n"}, "2201f12937b360b8a64b730b9ab5572701837f58": {"t": 1785767268.6861088, "n": 193624, "h": "ce2673e998f137f5", "c": "# Synthadoc — Design Document\n\n**Version:** 1.0.1 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n29. [Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer)\n30. [Per-Source Truncation Flag](#30-per-source-truncation-flag)\n31. [Proportional Context Budget](#31-proportional-context-budget)\n32. [Knowledge Graph](#32-knowledge-graph)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the full source text (bounded by `max_source_chars`) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `flag` / `skip`). Pages with `status='active'` are protected — sources that conflict with them trigger `flag` rather than `update` (RULE 1b).\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Orchestrator calls `_guard_ingest_cost()` on the returned cost: soft warn → `logger.warning` only; hard gate → `cost_gate_exceeded` audit event + `fail_permanent` (job → DEAD). On success, job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 0b — Key Data Extraction | None (regex) | Deterministic pre-processor: extracts numbers, percentages, currency amounts, ratios, formulas, underscore identifiers, and date ranges from source text via regex; appends them as a `[Key Data — extracted by pre-processor]` section before any LLM sees the text. Zero token cost; cannot hallucinate. Anchors the synthesis LLM to all quantitative candidates. |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads the full source text (bounded by `max_source_chars`) + BM25 candidates + `purpose.md` scope + status of each candidate page. Outputs per-page action: `create`, `update`, `flag`, `skip`. **RULE 1b:** a page with `status='active'` is human-reviewed and authoritative — when the source provides a conflicting value, date, formula, or conclusion, the action must be `flag`, not `update`. The only exception is a source that adds a topic the page does not yet mention at all. Decision cache version: `v2`. |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. If no citation markers are present in the returned body, a `citation_pass4_no_markers` WARNING audit event is written — the page is still saved but the condition is flagged for re-ingest with a more capable model. Zero-citation results are not cached so a subsequent re-ingest can succeed. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic. **Suppressed when TF fallback was used** (a small corpus with broad coverage may legitimately return fewer than 3 candidates — that is expected, not a gap).\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n| Citation presence (Check 5b) | Page body has ≥ 50 words and zero `^[filename:L-L]` citation markers → WARNING. Diagnostic only — does not block promotion to `active`. Indicates the annotation pass failed to produce markers, usually a model-compatibility issue (use Gemini 2.5 Flash or higher for reliable citation annotation). |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n### ActionAgent\n\nDispatches action-intent queries from the chat UI (e.g. \"activate a draft page\", \"show lint report\", \"archive a stale page\", \"run scaffold\"). Parses the user's free-text intent into a structured action via an LLM extraction prompt, then executes the action against the wiki and returns a human-readable result. When the intent is ambiguous (e.g. \"activate a page\" without specifying which), sets `needs_clarification=True` and returns a prompt and candidate list for the web UI to render as chip buttons.\n\n### ScaffoldAgent\n\nGenerates and updates the wiki's structural pages. Reads all current wiki pages and asks the LLM to organise them into 5–8 domain-appropriate categories, then writes `wiki/index.md` (category index with wikilinks), `wiki/purpose.md` (scope statement), and `AGENTS.md` (ingest and query guidelines). Also stamps a `categories:` field on each page's YAML frontmatter. When `ROUTING.md` already exists, regenerates it to stay in sync with the updated index structure.\n\n### RewriteAgent\n\nRewrites follow-up questions in multi-turn conversations into self-contained, standalone form before BM25 retrieval. Converts context-dependent phrases (\"What came after that?\", \"Tell me more about his early life\") into explicit questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the correct pages without relying on conversation context. Only invoked when conversation history is non-empty. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### SummarizeAgent\n\nCompresses the oldest conversation turns into a single `[Session summary: …]` assistant turn when a session exceeds `conversation_history_turns`. Prevents unbounded context growth across long sessions. Emits a `notice` SSE event the first time compression occurs so the user can see that earlier context was condensed. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### ContextAgent\n\nBuilds token-budgeted context packs for MCP tool consumers. See [Section 20 — Context Packs](#20-context-packs) for full detail.\n\n### SearchDecomposeAgent\n\nDecomposes a web search intent into 1–4 terse keyword search strings optimised for authoritative source retrieval. Uses a separate prompt from `QueryAgent`'s question decomposition — search decomposition asks \"what distinct search strings would find the best sources?\" (terse keyword phrases) rather than \"what distinct questions does this ask?\" (natural-language sub-questions). See [Web search fan-out](#web-search-fan-out) in the IngestAgent section.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n**Compound identifier expansion:** When a token contains underscores, the tokenizer emits both the compound form and each component as separate tokens (`capex_growth` → `[\"capex_growth\", \"capex\", \"growth\"]`). This expansion is applied at both index time and query time so that human-friendly queries match programmatic identifiers and vice versa. Leading/trailing underscores are stripped before splitting; no empty strings are emitted.\n\n**TF fallback for small corpora:** When all BM25 scores in a result set are zero or negative (IDF collapse on a corpus of ≤ 2 documents, or a term present in every page), the search system falls back to normalised term frequency (TF) scoring: `TF(doc, query) = Σcount(term, doc) / len(doc_tokens)`. Results are returned in the same format as BM25 results with a `tf_fallback: True` flag on each `SearchResult`. This field is recorded in the audit trail and consumed by gap detection (see below).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp}` |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n| `GET` | `/graph` _(v1.0.0)_ | — | `{status, node_count, edge_count, cluster_count, nodes: [...], edges: [...]}` or `{status: \"computing\"}` on first call |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 1.0.1\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n**Reading View default:** `synthadoc plugin install` and `synthadoc plugin upgrade` write `\"defaultViewMode\": \"preview\"` to `.obsidian/app.json` after copying the plugin files. This causes Obsidian to open new notes in Reading View by default, which is required for `^[filename:L-L]` citation chips to render. Only `defaultViewMode` is written; all other keys in `app.json` are preserved. If `app.json` is malformed JSON the write is skipped and a warning is logged — the install still succeeds.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports two things: dangling slugs (pages listed in ROUTING.md that no longer exist in the wiki) and unassigned slugs (pages that exist in index.md but are not listed in any ROUTING.md branch). Enabled only when ROUTING.md exists. **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [-w wiki] — generate ROUTING.md from index.md branch structure\n│ ├── validate [-w wiki] — report dangling slugs and cross-branch duplicates\n│ └── clean [-w wiki] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [-w wiki]\n├── candidates\n│ ├── list [-w wiki]\n│ ├── promote |--all [-w wiki]\n│ └── discard |--all [-w wiki]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Emit `logger.warning` in server log when per-job ingest cost exceeds this threshold; job continues normally |\n| `cost.hard_gate_usd` | float | `2.00` | Permanently fail the ingest job (DEAD) and record a `cost_gate_exceeded` audit event when per-job cost exceeds this threshold |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × (backup_count + 1)` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n| `ingest.max_source_chars` | int | `32000` | Character limit applied to each source before the LLM call. Sources exceeding this limit are truncated; the page's `sources:` frontmatter entry gets `truncated: true` and lint emits a warning. Override per-run with `--max-source-chars N`. _(v1.0.0)_ |\n| `query.context_wiki_pct` | float | `0.60` | Fraction of the model context window reserved for wiki source pages. _(v1.0.0)_ |\n| `query.context_history_pct` | float | `0.20` | Fraction reserved for conversation history. _(v1.0.0)_ |\n| `query.context_system_pct` | float | `0.15` | Fraction reserved for system prompt and instructions. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_index_pct` | float | `0.05` | Fraction reserved for the wiki index preamble. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_window` | int | _(auto)_ | Override the built-in context window lookup for the configured model. Use when running a local or custom model whose window size is not in the built-in table. _(v1.0.0)_ |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement is wired into the ingest pipeline: `Orchestrator._guard_ingest_cost()` is called after each ingest LLM call with the returned cost, before the job is marked complete.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | `logger.warning` in server log; job continues normally |\n| `hard_gate_usd` | $2.00 | Server/ingest path: records `cost_gate_exceeded` audit event + permanently fails job (DEAD). Interactive CLI: prompts `Proceed? [y/N]` |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` by default. The bind address is configurable via `server.host` in `config.toml` (e.g. `host = \"0.0.0.0\"` to expose on all interfaces for LAN access). No built-in authentication — restrict via firewall when exposing beyond loopback. Remote access without a reverse proxy is not recommended on shared or networked machines.\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs and unassigned slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `-w ` / `--wiki ` to target a specific wiki.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; a missing source file triggers `archived`. Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: if the source path recorded in frontmatter no longer exists on disk, the page is transitioned to `archived`.\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 5) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 5 # turns to include in each request (default: 5; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp}` or `{error}` | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── hooks/ ← user hook scripts (always included)\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ ├── extracted/ ← text sidecars + PDF pagemaps (always included)\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `logs/`.\n\n> **v0.9.3 fix:** `hooks/` and `.synthadoc/extracted/` were previously missing from backups. Omitting `extracted/` caused the Source Viewer and Provenance citation modal to show \"Could not read source file\" after restore.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## 29. Pre-LLM Source Sanitizer\n\nEvery source document passes through a sanitizer immediately after text extraction, before any LLM call. The sanitizer removes six categories of content that could manipulate the LLM or degrade compilation quality:\n\n| Category | Action | Warning logged? |\n|---|---|---|\n| Zero-width characters (U+200B, U+200C, U+200D, U+FEFF) | Removed silently | No |\n| Bidi override characters (U+202A–U+202E, U+2066–U+2069) | Removed | Yes |\n| HTML comments (``) | Removed silently | No |\n| Hidden CSS spans (`display:none`, `visibility:hidden`) | Removed silently | No |\n| Base64 blobs ≥ 200 consecutive characters | Replaced with `[base64 content removed]` | Yes |\n| Instruction-override phrases (8 patterns: \"ignore previous instructions\", \"disregard the above\", \"override your system prompt\", etc.) | Replaced with `[redacted]` | Always |\n\nWhen content is removed and a warning is appropriate, the server logs a `WARN` entry:\n```\n[WARN] sanitizer stripped content from 'papers/survey.pdf': bidi overrides, instruction-override phrase\n```\n\nThis is a pure pre-processing step with zero LLM cost. Truncation (if configured) is applied after sanitization, so the character budget is consumed by clean text only.\n\n---\n\n## 30. Per-Source Truncation Flag\n\nLarge source documents are truncated before they reach the LLM. By default the limit is **32,000 characters** (~8,000 tokens), raised from the prior hardcoded 8,000-character limit. When a source exceeds this limit, its compiled wiki page records a `truncated: true` flag in its `sources:` frontmatter entry.\n\n### Configuration\n\n```toml\n[ingest]\nmax_source_chars = 32000 # default; raise for large PDFs or books\n```\n\n### Per-ingest override\n\nThe limit can be raised for a single ingest run without modifying the config:\n\n```bash\nsynthadoc ingest papers/large-textbook.pdf --max-source-chars 128000\n```\n\nThe same `max_source_chars` field is accepted by the HTTP `POST /jobs/ingest` body and the MCP `synthadoc_ingest` tool.\n\n### Lint surfacing\n\n`synthadoc lint` emits a warning for any page that has a truncated source:\n\n```\n[WARN] quantum-computing.md: source 'papers/large-survey.pdf' was truncated at ingest\n (source exceeded max_source_chars=32000 — 87,412 chars in source).\n To re-ingest with a higher limit (this source only):\n synthadoc ingest papers/large-survey.pdf --max-source-chars 128000\n To raise the limit for all future ingests:\n set [ingest] max_source_chars = 128000 in your config\n```\n\n---\n\n## 31. Proportional Context Budget\n\nQuery context is allocated proportionally to the configured model's context window, replacing the prior hard cap. The default allocation:\n\n| Slice | Default | Purpose |\n|---|---|---|\n| `context_wiki_pct` | 60% | Wiki page content (ranked by BM25 + vector score) |\n| `context_history_pct` | 20% | Chat turn history (newest-first, oldest dropped when over budget) |\n| `context_system_pct` | 15% | System prompt and purpose document |\n| `context_index_pct` | 5% | `ROUTING.md` and search index |\n\nPercentages must sum to ≤ 100. Configure in `[query]`:\n\n```toml\n[query]\ncontext_window = 0 # 0 = auto-detect from known model map\ncontext_wiki_pct = 60\ncontext_history_pct = 20\ncontext_system_pct = 15\ncontext_index_pct = 5\n```\n\n> **Note:** `context_system_pct` and `context_index_pct` are parsed and validated but not yet enforced as hard caps in this release — the system prompt uses a conservative fixed limit, which is well within the configured 15% slice for all supported models. Full per-slice enforcement is planned for v1.1.\n\n### Model context window map\n\nSynthadoc ships a built-in prefix-matched table:\n\n| Model | Context window |\n|---|---|\n| `claude-opus-4*`, `claude-sonnet-4*`, `claude-haiku-4*` | 200,000 tokens |\n| `gpt-4o*`, `gpt-4-turbo*` | 128,000 tokens |\n| `gpt-4` (exact) | 8,192 tokens |\n| `gpt-3.5-turbo*` | 16,385 tokens |\n| Unknown / fallback | 128,000 tokens |\n\nSet `context_window = N` in config to override the map (useful for local Ollama models with non-standard context sizes).\n\nAt the defaults on a 200k-token model: wiki slice ≈ 480,000 chars — far more than any practical query needs. On constrained models, the budget scales down automatically so the same config works across model families.\n\n---\n\n## 32. Knowledge Graph\n\nSynthadoc computes a wikilink graph across all active and draft pages during every lint run. The graph is stored in two tables in `audit.db`:\n\n```sql\nCREATE TABLE graph_nodes (\n slug TEXT PRIMARY KEY,\n cluster_id INTEGER NOT NULL,\n updated_at TEXT NOT NULL\n);\n\nCREATE TABLE graph_edges (\n from_slug TEXT NOT NULL,\n to_slug TEXT NOT NULL,\n weight INTEGER NOT NULL DEFAULT 1,\n PRIMARY KEY (from_slug, to_slug)\n);\n```\n\nEdge weight equals the number of `[[wikilink]]` references from one page to another (most pairs: 1). Cluster IDs are assigned by Louvain community detection — pages that link densely to each other end up in the same cluster.\n\n### REST API\n\n**`GET /graph`** — returns the full graph with per-node enrichment:\n\n```json\n{\n \"status\": \"ready\",\n \"node_count\": 45,\n \"edge_count\": 123,\n \"cluster_count\": 4,\n \"nodes\": [\n {\"slug\": \"quantum-error-correction\", \"title\": \"Quantum Error Correction\",\n \"type\": \"concept\", \"state\": \"active\", \"cluster_id\": 2}\n ],\n \"edges\": [\n {\"from\": \"quantum-error-correction\", \"to\": \"shor-algorithm\", \"weight\": 1}\n ]\n}\n```\n\nWhen the graph tables are empty (first call after upgrade, or before the first lint run), the server returns `{\"status\": \"computing\"}` and starts a background build immediately. The web UI polls every 2 seconds and renders the graph when the status becomes `\"ready\"`.\n\n### Web UI\n\nThe **Graph** tab (alongside Chat in the top nav) renders a D3.js force-directed graph. Nodes are colored by cluster. Clicking a node opens a sidebar showing the page title, type and lifecycle state badges, cluster assignment, and an **\"Ask about this →\"** button that switches to the Chat tab with a pre-filled query.\n\nControls: zoom in/out (scroll or pinch), drag nodes, filter by page type.\n\n---\n\n## Customization\n\nSynthadoc exposes four extension points — all hot-loaded, no server restart required.\n\n### Custom skills (new file formats)\n\nSubclass `BaseSkill` (Apache-2.0 — no AGPL obligation on your skill code), drop the folder in `/skills/` or `~/.synthadoc/skills/`, and Synthadoc picks it up on the next ingest. Skills match by file extension or intent prefix and support any Unicode text, including CJK prefixes.\n\n→ Full interface, manifest format, and examples: [§5 Skills System](#5-skills-system) and [§17 Plugin Development Guide](#17-plugin-development-guide)\n\n### Custom LLM providers\n\nSubclass `LLMProvider` from `synthadoc/providers/base.py` (Apache-2.0) and place the file in `~/.synthadoc/providers/` or the wiki `providers/` directory. Switch active provider with one config line.\n\n→ Full interface and wiring instructions: [§10 Configuration — Provider switching](#10-configuration)\n\n### Hooks\n\nShell scripts (any language) that fire on `on_ingest_complete` and `on_lint_complete`. Receive a JSON context on stdin. Set `blocking = true` to gate the operation on the hook's exit code — useful for CI pipelines and quality gates.\n\n→ Full event schema, context JSON, and blocking behaviour: [§11 Hook System](#11-hook-system)\n\n### Cache control\n\nThree cache layers (embedding, LLM response, provider prompt cache) invalidate automatically on source-file change (SHA-256). Force a fresh call with `--force`, or wipe all cached responses with `synthadoc cache clear -w `.\n\n→ Layer-by-layer breakdown and invalidation rules: [§12 Cache System](#12-cache-system)\n\n### Per-wiki AGENTS.md\n\nEdit `/AGENTS.md` to give the LLM domain-specific instructions — terminology, page-naming conventions, what to cross-reference. This is the highest-priority instruction source for every agent run against this wiki; the `scaffold` command regenerates a starter version from current wiki state.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v1.0.1\n\n- **Scaffold JSON extraction hardened** — replaced the greedy regex used to extract the LLM's JSON response with a brace-balanced parser; eliminates false-positive matches on `[[wikilink]]` syntax that previously caused `json.loads()` failures and silent scaffold corruption\n- **Generated file validation** — scaffold now validates every LLM-generated file (index, purpose, AGENTS.md) for minimum length, required frontmatter, and `[[wikilink]]` presence before writing to disk; validation failure marks the job as failed with a structured error message rather than silently writing a malformed page\n- **ROUTING.md regeneration validation** — `routing init` and scheduled routing regeneration now validate the regenerated file for minimum line count and anchor presence before applying it; prevents a corrupt or truncated ROUTING.md from entering the wiki\n- **Scaffold failure reporting** — scaffold skip and failure reasons are now surfaced in the terminal output and job error field; previously silent failures (e.g. LLM timeout, JSON parse error) are now visible without tailing the JSON log\n- **Install-time scaffold removed** — `synthadoc install` no longer runs the LLM scaffold call during installation; scaffold is a separate step (`synthadoc scaffold`) that the user runs after setting their API key, avoiding confusing API-key-not-set errors at install time\n- **AquaFlow LLM evaluation** — 15-question M&A due-diligence benchmark across five models (MiniMax-Think M3, Claude Opus 4.8, Claude Sonnet 4.6, DeepSeek-R1, Qwen Plus); full report at `docs/example/aquaflow/evaluation/report/llm-query-benchmark.md`\n\n### v1.0.0\n\n- **Pre-LLM source sanitizer** — every source document passes through a sanitizer before any LLM call; removes six categories of potentially harmful content (zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars, instruction-override phrases); a `WARN` log entry is emitted when bidi overrides or instruction-override phrases are stripped; zero LLM cost; applied before truncation so the character budget is consumed by clean text only\n- **Per-source truncation flag** — default character limit raised from 8,000 to 32,000 characters per source (~8,000 tokens); truncated sources are flagged with `truncated: true` in the page's `sources:` frontmatter; `--max-source-chars N` CLI flag and matching `POST /jobs/ingest` body field and MCP `synthadoc_ingest` parameter override the limit per run; `synthadoc lint` emits a warning for any page with a truncated source and suggests the override command; configurable via `[ingest] max_source_chars` in `config.toml`\n- **Proportional context budget** — query context is allocated proportionally to the configured model's context window; four configurable slices: `context_wiki_pct` (60%), `context_history_pct` (20%), `context_system_pct` (15%), `context_index_pct` (5%); built-in prefix-matched model context window table covers Claude 4, GPT-4o/turbo/3.5, and GPT-4; unknown models fall back to 128,000 tokens; `context_window = N` in `[query]` overrides the table for local or custom models\n- **Knowledge graph** — wikilink graph computed across all active and draft pages during every lint run; stored in `graph_nodes` and `graph_edges` tables in `audit.db`; edge weight = number of `[[wikilink]]` references between two pages; cluster IDs assigned by Louvain community detection; `GET /graph` REST endpoint returns nodes with per-node enrichment (title, type, state, cluster) and edges; first call after upgrade triggers a background build and returns `{\"status\":\"computing\"}`; web UI **Graph** tab (D3.js force-directed graph) — nodes colored by cluster, click to see page details and an \"Ask about this →\" button that opens a pre-filled chat query\n- **Citation accuracy & rendering improvements** — six correctness fixes to the ingest citation pipeline: (1) full source text passed to the decision LLM so citations can reference any line in the document; (2) `_annotate_citations` bugs A–F resolved (false-positive matches, off-by-one line ranges, duplicate markers, escaped bracket handling, multi-paragraph span errors, and source filename normalisation); (3) `_has_citations` check added — lint warns and suggests a model upgrade when a page has zero citation markers after synthesis; (4) `Key Data` section extraction hardened so numerical facts, rates, and formulas are preserved verbatim across the annotation pass; (5) `obsidianCitationsToGfm` rendering fixed in the web UI — citation superscripts now display correctly in the chat panel alongside streamed answers; (6) reading view set as the default Obsidian display mode for wiki pages\n\n### v0.9.3\n\n- **Backup now includes `hooks/` and `.synthadoc/extracted/`** — both directories were silently omitted from backups. `hooks/` contains user-customised hook scripts; `.synthadoc/extracted/` contains per-source text sidecars and PDF pagemaps written by the ingest agent. Without these, the Source Viewer and Provenance citation modal showed \"Could not read source file\" after restore.\n- **PyPI publish workflow** — `publish.yml` triggers on GitHub Release, builds the wheel with hatchling, and uploads to PyPI via OIDC trusted publishing (no API token stored). `pip install synthadoc` now works.\n- **Web UI bundled in pip package** — `synthadoc/data/web-ui/dist/` is now included in the wheel. `synthadoc web` previously returned 503 on pip-installed instances because the React dist was not packaged.\n- **CI sync checks** — PRs now fail if `synthadoc/data/obsidian-plugin/`, `synthadoc/data/web-ui/`, or `README-pypi.md` are out of sync with their source. All sync-check steps use `shell: bash` for Windows runner compatibility.\n- **`bump_version.py` extended** — now also updates `synthadoc/data/obsidian-plugin/manifest.json` alongside the source manifest, preventing version drift between the bundled copy and the plugin source.\n\n### v0.9.2\n\n- **`synthadoc/data/web-ui/dist/` packaged** — web UI dist files committed to the repo and bundled in the wheel via hatchling artifacts.\n- **`scripts/sync_web_ui.py`** — helper script to sync `web-ui/dist/` into `synthadoc/data/web-ui/dist/` after `npm run build`.\n\n### v0.9.1\n\n- **PyPI metadata** — added `readme`, `keywords`, `classifiers`, and `[project.urls]` to `pyproject.toml` for correct PyPI rendering.\n- **`README-pypi.md`** — generated from `README.md` by `scripts/prepare_pypi_readme.py`; strips `pypi-strip` blocks and rewrites relative links to absolute GitHub URLs.\n- **Branch protection compatibility** — removed post-merge `build-plugin` CI job that pushed compiled plugin directly to `main`. Plugin sync is now enforced in PRs: CI fails if `synthadoc/data/obsidian-plugin/` is out of sync.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 5). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n"}, "94a2fe7743b98bf8c9557d5e3e19e52170883e9e": {"t": 1785767270.9531944, "n": 193338, "h": "0d79514cbbd23aee", "c": "# Synthadoc — Design Document\n\n**Version:** 1.0.1 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n29. [Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer)\n30. [Per-Source Truncation Flag](#30-per-source-truncation-flag)\n31. [Proportional Context Budget](#31-proportional-context-budget)\n32. [Knowledge Graph](#32-knowledge-graph)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent, checks CostGuard\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the full source text (bounded by `max_source_chars`) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `flag` / `skip`). Pages with `status='active'` are protected — sources that conflict with them trigger `flag` rather than `update` (RULE 1b).\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 0b — Key Data Extraction | None (regex) | Deterministic pre-processor: extracts numbers, percentages, currency amounts, ratios, formulas, underscore identifiers, and date ranges from source text via regex; appends them as a `[Key Data — extracted by pre-processor]` section before any LLM sees the text. Zero token cost; cannot hallucinate. Anchors the synthesis LLM to all quantitative candidates. |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads the full source text (bounded by `max_source_chars`) + BM25 candidates + `purpose.md` scope + status of each candidate page. Outputs per-page action: `create`, `update`, `flag`, `skip`. **RULE 1b:** a page with `status='active'` is human-reviewed and authoritative — when the source provides a conflicting value, date, formula, or conclusion, the action must be `flag`, not `update`. The only exception is a source that adds a topic the page does not yet mention at all. Decision cache version: `v2`. |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. If no citation markers are present in the returned body, a `citation_pass4_no_markers` WARNING audit event is written — the page is still saved but the condition is flagged for re-ingest with a more capable model. Zero-citation results are not cached so a subsequent re-ingest can succeed. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic. **Suppressed when TF fallback was used** (a small corpus with broad coverage may legitimately return fewer than 3 candidates — that is expected, not a gap).\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n| Citation presence (Check 5b) | Page body has ≥ 50 words and zero `^[filename:L-L]` citation markers → WARNING. Diagnostic only — does not block promotion to `active`. Indicates the annotation pass failed to produce markers, usually a model-compatibility issue (use Gemini 2.5 Flash or higher for reliable citation annotation). |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n### ActionAgent\n\nDispatches action-intent queries from the chat UI (e.g. \"activate a draft page\", \"show lint report\", \"archive a stale page\", \"run scaffold\"). Parses the user's free-text intent into a structured action via an LLM extraction prompt, then executes the action against the wiki and returns a human-readable result. When the intent is ambiguous (e.g. \"activate a page\" without specifying which), sets `needs_clarification=True` and returns a prompt and candidate list for the web UI to render as chip buttons.\n\n### ScaffoldAgent\n\nGenerates and updates the wiki's structural pages. Reads all current wiki pages and asks the LLM to organise them into 5–8 domain-appropriate categories, then writes `wiki/index.md` (category index with wikilinks), `wiki/purpose.md` (scope statement), and `AGENTS.md` (ingest and query guidelines). Also stamps a `categories:` field on each page's YAML frontmatter. When `ROUTING.md` already exists, regenerates it to stay in sync with the updated index structure.\n\n### RewriteAgent\n\nRewrites follow-up questions in multi-turn conversations into self-contained, standalone form before BM25 retrieval. Converts context-dependent phrases (\"What came after that?\", \"Tell me more about his early life\") into explicit questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the correct pages without relying on conversation context. Only invoked when conversation history is non-empty. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### SummarizeAgent\n\nCompresses the oldest conversation turns into a single `[Session summary: …]` assistant turn when a session exceeds `conversation_history_turns`. Prevents unbounded context growth across long sessions. Emits a `notice` SSE event the first time compression occurs so the user can see that earlier context was condensed. See also: [Multi-turn Conversation](#multi-turn-conversation).\n\n### ContextAgent\n\nBuilds token-budgeted context packs for MCP tool consumers. See [Section 20 — Context Packs](#20-context-packs) for full detail.\n\n### SearchDecomposeAgent\n\nDecomposes a web search intent into 1–4 terse keyword search strings optimised for authoritative source retrieval. Uses a separate prompt from `QueryAgent`'s question decomposition — search decomposition asks \"what distinct search strings would find the best sources?\" (terse keyword phrases) rather than \"what distinct questions does this ask?\" (natural-language sub-questions). See [Web search fan-out](#web-search-fan-out) in the IngestAgent section.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n**Compound identifier expansion:** When a token contains underscores, the tokenizer emits both the compound form and each component as separate tokens (`capex_growth` → `[\"capex_growth\", \"capex\", \"growth\"]`). This expansion is applied at both index time and query time so that human-friendly queries match programmatic identifiers and vice versa. Leading/trailing underscores are stripped before splitting; no empty strings are emitted.\n\n**TF fallback for small corpora:** When all BM25 scores in a result set are zero or negative (IDF collapse on a corpus of ≤ 2 documents, or a term present in every page), the search system falls back to normalised term frequency (TF) scoring: `TF(doc, query) = Σcount(term, doc) / len(doc_tokens)`. Results are returned in the same format as BM25 results with a `tf_fallback: True` flag on each `SearchResult`. This field is recorded in the audit trail and consumed by gap detection (see below).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp}` |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n| `GET` | `/graph` _(v1.0.0)_ | — | `{status, node_count, edge_count, cluster_count, nodes: [...], edges: [...]}` or `{status: \"computing\"}` on first call |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 1.0.1\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n**Reading View default:** `synthadoc plugin install` and `synthadoc plugin upgrade` write `\"defaultViewMode\": \"preview\"` to `.obsidian/app.json` after copying the plugin files. This causes Obsidian to open new notes in Reading View by default, which is required for `^[filename:L-L]` citation chips to render. Only `defaultViewMode` is written; all other keys in `app.json` are preserved. If `app.json` is malformed JSON the write is skipped and a warning is logged — the install still succeeds.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports two things: dangling slugs (pages listed in ROUTING.md that no longer exist in the wiki) and unassigned slugs (pages that exist in index.md but are not listed in any ROUTING.md branch). Enabled only when ROUTING.md exists. **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [-w wiki] — generate ROUTING.md from index.md branch structure\n│ ├── validate [-w wiki] — report dangling slugs and cross-branch duplicates\n│ └── clean [-w wiki] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [-w wiki]\n├── candidates\n│ ├── list [-w wiki]\n│ ├── promote |--all [-w wiki]\n│ └── discard |--all [-w wiki]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Log warning, continue _(configured but not yet enforced — cost_guard is wired to the config but check() is not called in the ingest path)_ |\n| `cost.hard_gate_usd` | float | `2.00` | Require explicit confirmation _(configured but not yet enforced — see above)_ |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × (backup_count + 1)` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n| `ingest.max_source_chars` | int | `32000` | Character limit applied to each source before the LLM call. Sources exceeding this limit are truncated; the page's `sources:` frontmatter entry gets `truncated: true` and lint emits a warning. Override per-run with `--max-source-chars N`. _(v1.0.0)_ |\n| `query.context_wiki_pct` | float | `0.60` | Fraction of the model context window reserved for wiki source pages. _(v1.0.0)_ |\n| `query.context_history_pct` | float | `0.20` | Fraction reserved for conversation history. _(v1.0.0)_ |\n| `query.context_system_pct` | float | `0.15` | Fraction reserved for system prompt and instructions. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_index_pct` | float | `0.05` | Fraction reserved for the wiki index preamble. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_window` | int | _(auto)_ | Override the built-in context window lookup for the configured model. Use when running a local or custom model whose window size is not in the built-in table. _(v1.0.0)_ |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python hooks/git-auto-commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement (soft warn / hard gate) is implemented and configurable but not yet wired into the LLM call path — it is infrastructure ready to activate.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | Log warning; auto-continue |\n| `hard_gate_usd` | $2.00 | Prompt user `Proceed? [y/N]`; block if N; skip prompt if `auto_confirm=True` or `--yes` flag |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` by default. The bind address is configurable via `server.host` in `config.toml` (e.g. `host = \"0.0.0.0\"` to expose on all interfaces for LAN access). No built-in authentication — restrict via firewall when exposing beyond loopback. Remote access without a reverse proxy is not recommended on shared or networked machines.\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs and unassigned slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `-w ` / `--wiki ` to target a specific wiki.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; a missing source file triggers `archived`. Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: if the source path recorded in frontmatter no longer exists on disk, the page is transitioned to `archived`.\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 5) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 5 # turns to include in each request (default: 5; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp}` or `{error}` | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── hooks/ ← user hook scripts (always included)\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ ├── extracted/ ← text sidecars + PDF pagemaps (always included)\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `logs/`.\n\n> **v0.9.3 fix:** `hooks/` and `.synthadoc/extracted/` were previously missing from backups. Omitting `extracted/` caused the Source Viewer and Provenance citation modal to show \"Could not read source file\" after restore.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## 29. Pre-LLM Source Sanitizer\n\nEvery source document passes through a sanitizer immediately after text extraction, before any LLM call. The sanitizer removes six categories of content that could manipulate the LLM or degrade compilation quality:\n\n| Category | Action | Warning logged? |\n|---|---|---|\n| Zero-width characters (U+200B, U+200C, U+200D, U+FEFF) | Removed silently | No |\n| Bidi override characters (U+202A–U+202E, U+2066–U+2069) | Removed | Yes |\n| HTML comments (``) | Removed silently | No |\n| Hidden CSS spans (`display:none`, `visibility:hidden`) | Removed silently | No |\n| Base64 blobs ≥ 200 consecutive characters | Replaced with `[base64 content removed]` | Yes |\n| Instruction-override phrases (8 patterns: \"ignore previous instructions\", \"disregard the above\", \"override your system prompt\", etc.) | Replaced with `[redacted]` | Always |\n\nWhen content is removed and a warning is appropriate, the server logs a `WARN` entry:\n```\n[WARN] sanitizer stripped content from 'papers/survey.pdf': bidi overrides, instruction-override phrase\n```\n\nThis is a pure pre-processing step with zero LLM cost. Truncation (if configured) is applied after sanitization, so the character budget is consumed by clean text only.\n\n---\n\n## 30. Per-Source Truncation Flag\n\nLarge source documents are truncated before they reach the LLM. By default the limit is **32,000 characters** (~8,000 tokens), raised from the prior hardcoded 8,000-character limit. When a source exceeds this limit, its compiled wiki page records a `truncated: true` flag in its `sources:` frontmatter entry.\n\n### Configuration\n\n```toml\n[ingest]\nmax_source_chars = 32000 # default; raise for large PDFs or books\n```\n\n### Per-ingest override\n\nThe limit can be raised for a single ingest run without modifying the config:\n\n```bash\nsynthadoc ingest papers/large-textbook.pdf --max-source-chars 128000\n```\n\nThe same `max_source_chars` field is accepted by the HTTP `POST /jobs/ingest` body and the MCP `synthadoc_ingest` tool.\n\n### Lint surfacing\n\n`synthadoc lint` emits a warning for any page that has a truncated source:\n\n```\n[WARN] quantum-computing.md: source 'papers/large-survey.pdf' was truncated at ingest\n (source exceeded max_source_chars=32000 — 87,412 chars in source).\n To re-ingest with a higher limit (this source only):\n synthadoc ingest papers/large-survey.pdf --max-source-chars 128000\n To raise the limit for all future ingests:\n set [ingest] max_source_chars = 128000 in your config\n```\n\n---\n\n## 31. Proportional Context Budget\n\nQuery context is allocated proportionally to the configured model's context window, replacing the prior hard cap. The default allocation:\n\n| Slice | Default | Purpose |\n|---|---|---|\n| `context_wiki_pct` | 60% | Wiki page content (ranked by BM25 + vector score) |\n| `context_history_pct` | 20% | Chat turn history (newest-first, oldest dropped when over budget) |\n| `context_system_pct` | 15% | System prompt and purpose document |\n| `context_index_pct` | 5% | `ROUTING.md` and search index |\n\nPercentages must sum to ≤ 100. Configure in `[query]`:\n\n```toml\n[query]\ncontext_window = 0 # 0 = auto-detect from known model map\ncontext_wiki_pct = 60\ncontext_history_pct = 20\ncontext_system_pct = 15\ncontext_index_pct = 5\n```\n\n> **Note:** `context_system_pct` and `context_index_pct` are parsed and validated but not yet enforced as hard caps in this release — the system prompt uses a conservative fixed limit, which is well within the configured 15% slice for all supported models. Full per-slice enforcement is planned for v1.1.\n\n### Model context window map\n\nSynthadoc ships a built-in prefix-matched table:\n\n| Model | Context window |\n|---|---|\n| `claude-opus-4*`, `claude-sonnet-4*`, `claude-haiku-4*` | 200,000 tokens |\n| `gpt-4o*`, `gpt-4-turbo*` | 128,000 tokens |\n| `gpt-4` (exact) | 8,192 tokens |\n| `gpt-3.5-turbo*` | 16,385 tokens |\n| Unknown / fallback | 128,000 tokens |\n\nSet `context_window = N` in config to override the map (useful for local Ollama models with non-standard context sizes).\n\nAt the defaults on a 200k-token model: wiki slice ≈ 480,000 chars — far more than any practical query needs. On constrained models, the budget scales down automatically so the same config works across model families.\n\n---\n\n## 32. Knowledge Graph\n\nSynthadoc computes a wikilink graph across all active and draft pages during every lint run. The graph is stored in two tables in `audit.db`:\n\n```sql\nCREATE TABLE graph_nodes (\n slug TEXT PRIMARY KEY,\n cluster_id INTEGER NOT NULL,\n updated_at TEXT NOT NULL\n);\n\nCREATE TABLE graph_edges (\n from_slug TEXT NOT NULL,\n to_slug TEXT NOT NULL,\n weight INTEGER NOT NULL DEFAULT 1,\n PRIMARY KEY (from_slug, to_slug)\n);\n```\n\nEdge weight equals the number of `[[wikilink]]` references from one page to another (most pairs: 1). Cluster IDs are assigned by Louvain community detection — pages that link densely to each other end up in the same cluster.\n\n### REST API\n\n**`GET /graph`** — returns the full graph with per-node enrichment:\n\n```json\n{\n \"status\": \"ready\",\n \"node_count\": 45,\n \"edge_count\": 123,\n \"cluster_count\": 4,\n \"nodes\": [\n {\"slug\": \"quantum-error-correction\", \"title\": \"Quantum Error Correction\",\n \"type\": \"concept\", \"state\": \"active\", \"cluster_id\": 2}\n ],\n \"edges\": [\n {\"from\": \"quantum-error-correction\", \"to\": \"shor-algorithm\", \"weight\": 1}\n ]\n}\n```\n\nWhen the graph tables are empty (first call after upgrade, or before the first lint run), the server returns `{\"status\": \"computing\"}` and starts a background build immediately. The web UI polls every 2 seconds and renders the graph when the status becomes `\"ready\"`.\n\n### Web UI\n\nThe **Graph** tab (alongside Chat in the top nav) renders a D3.js force-directed graph. Nodes are colored by cluster. Clicking a node opens a sidebar showing the page title, type and lifecycle state badges, cluster assignment, and an **\"Ask about this →\"** button that switches to the Chat tab with a pre-filled query.\n\nControls: zoom in/out (scroll or pinch), drag nodes, filter by page type.\n\n---\n\n## Customization\n\nSynthadoc exposes four extension points — all hot-loaded, no server restart required.\n\n### Custom skills (new file formats)\n\nSubclass `BaseSkill` (Apache-2.0 — no AGPL obligation on your skill code), drop the folder in `/skills/` or `~/.synthadoc/skills/`, and Synthadoc picks it up on the next ingest. Skills match by file extension or intent prefix and support any Unicode text, including CJK prefixes.\n\n→ Full interface, manifest format, and examples: [§5 Skills System](#5-skills-system) and [§17 Plugin Development Guide](#17-plugin-development-guide)\n\n### Custom LLM providers\n\nSubclass `LLMProvider` from `synthadoc/providers/base.py` (Apache-2.0) and place the file in `~/.synthadoc/providers/` or the wiki `providers/` directory. Switch active provider with one config line.\n\n→ Full interface and wiring instructions: [§10 Configuration — Provider switching](#10-configuration)\n\n### Hooks\n\nShell scripts (any language) that fire on `on_ingest_complete` and `on_lint_complete`. Receive a JSON context on stdin. Set `blocking = true` to gate the operation on the hook's exit code — useful for CI pipelines and quality gates.\n\n→ Full event schema, context JSON, and blocking behaviour: [§11 Hook System](#11-hook-system)\n\n### Cache control\n\nThree cache layers (embedding, LLM response, provider prompt cache) invalidate automatically on source-file change (SHA-256). Force a fresh call with `--force`, or wipe all cached responses with `synthadoc cache clear -w `.\n\n→ Layer-by-layer breakdown and invalidation rules: [§12 Cache System](#12-cache-system)\n\n### Per-wiki AGENTS.md\n\nEdit `/AGENTS.md` to give the LLM domain-specific instructions — terminology, page-naming conventions, what to cross-reference. This is the highest-priority instruction source for every agent run against this wiki; the `scaffold` command regenerates a starter version from current wiki state.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v1.0.1\n\n- **Scaffold JSON extraction hardened** — replaced the greedy regex used to extract the LLM's JSON response with a brace-balanced parser; eliminates false-positive matches on `[[wikilink]]` syntax that previously caused `json.loads()` failures and silent scaffold corruption\n- **Generated file validation** — scaffold now validates every LLM-generated file (index, purpose, AGENTS.md) for minimum length, required frontmatter, and `[[wikilink]]` presence before writing to disk; validation failure marks the job as failed with a structured error message rather than silently writing a malformed page\n- **ROUTING.md regeneration validation** — `routing init` and scheduled routing regeneration now validate the regenerated file for minimum line count and anchor presence before applying it; prevents a corrupt or truncated ROUTING.md from entering the wiki\n- **Scaffold failure reporting** — scaffold skip and failure reasons are now surfaced in the terminal output and job error field; previously silent failures (e.g. LLM timeout, JSON parse error) are now visible without tailing the JSON log\n- **Install-time scaffold removed** — `synthadoc install` no longer runs the LLM scaffold call during installation; scaffold is a separate step (`synthadoc scaffold`) that the user runs after setting their API key, avoiding confusing API-key-not-set errors at install time\n- **AquaFlow LLM evaluation** — 15-question M&A due-diligence benchmark across five models (MiniMax-Think M3, Claude Opus 4.8, Claude Sonnet 4.6, DeepSeek-R1, Qwen Plus); full report at `docs/example/aquaflow/evaluation/report/llm-query-benchmark.md`\n\n### v1.0.0\n\n- **Pre-LLM source sanitizer** — every source document passes through a sanitizer before any LLM call; removes six categories of potentially harmful content (zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars, instruction-override phrases); a `WARN` log entry is emitted when bidi overrides or instruction-override phrases are stripped; zero LLM cost; applied before truncation so the character budget is consumed by clean text only\n- **Per-source truncation flag** — default character limit raised from 8,000 to 32,000 characters per source (~8,000 tokens); truncated sources are flagged with `truncated: true` in the page's `sources:` frontmatter; `--max-source-chars N` CLI flag and matching `POST /jobs/ingest` body field and MCP `synthadoc_ingest` parameter override the limit per run; `synthadoc lint` emits a warning for any page with a truncated source and suggests the override command; configurable via `[ingest] max_source_chars` in `config.toml`\n- **Proportional context budget** — query context is allocated proportionally to the configured model's context window; four configurable slices: `context_wiki_pct` (60%), `context_history_pct` (20%), `context_system_pct` (15%), `context_index_pct` (5%); built-in prefix-matched model context window table covers Claude 4, GPT-4o/turbo/3.5, and GPT-4; unknown models fall back to 128,000 tokens; `context_window = N` in `[query]` overrides the table for local or custom models\n- **Knowledge graph** — wikilink graph computed across all active and draft pages during every lint run; stored in `graph_nodes` and `graph_edges` tables in `audit.db`; edge weight = number of `[[wikilink]]` references between two pages; cluster IDs assigned by Louvain community detection; `GET /graph` REST endpoint returns nodes with per-node enrichment (title, type, state, cluster) and edges; first call after upgrade triggers a background build and returns `{\"status\":\"computing\"}`; web UI **Graph** tab (D3.js force-directed graph) — nodes colored by cluster, click to see page details and an \"Ask about this →\" button that opens a pre-filled chat query\n- **Citation accuracy & rendering improvements** — six correctness fixes to the ingest citation pipeline: (1) full source text passed to the decision LLM so citations can reference any line in the document; (2) `_annotate_citations` bugs A–F resolved (false-positive matches, off-by-one line ranges, duplicate markers, escaped bracket handling, multi-paragraph span errors, and source filename normalisation); (3) `_has_citations` check added — lint warns and suggests a model upgrade when a page has zero citation markers after synthesis; (4) `Key Data` section extraction hardened so numerical facts, rates, and formulas are preserved verbatim across the annotation pass; (5) `obsidianCitationsToGfm` rendering fixed in the web UI — citation superscripts now display correctly in the chat panel alongside streamed answers; (6) reading view set as the default Obsidian display mode for wiki pages\n\n### v0.9.3\n\n- **Backup now includes `hooks/` and `.synthadoc/extracted/`** — both directories were silently omitted from backups. `hooks/` contains user-customised hook scripts; `.synthadoc/extracted/` contains per-source text sidecars and PDF pagemaps written by the ingest agent. Without these, the Source Viewer and Provenance citation modal showed \"Could not read source file\" after restore.\n- **PyPI publish workflow** — `publish.yml` triggers on GitHub Release, builds the wheel with hatchling, and uploads to PyPI via OIDC trusted publishing (no API token stored). `pip install synthadoc` now works.\n- **Web UI bundled in pip package** — `synthadoc/data/web-ui/dist/` is now included in the wheel. `synthadoc web` previously returned 503 on pip-installed instances because the React dist was not packaged.\n- **CI sync checks** — PRs now fail if `synthadoc/data/obsidian-plugin/`, `synthadoc/data/web-ui/`, or `README-pypi.md` are out of sync with their source. All sync-check steps use `shell: bash` for Windows runner compatibility.\n- **`bump_version.py` extended** — now also updates `synthadoc/data/obsidian-plugin/manifest.json` alongside the source manifest, preventing version drift between the bundled copy and the plugin source.\n\n### v0.9.2\n\n- **`synthadoc/data/web-ui/dist/` packaged** — web UI dist files committed to the repo and bundled in the wheel via hatchling artifacts.\n- **`scripts/sync_web_ui.py`** — helper script to sync `web-ui/dist/` into `synthadoc/data/web-ui/dist/` after `npm run build`.\n\n### v0.9.1\n\n- **PyPI metadata** — added `readme`, `keywords`, `classifiers`, and `[project.urls]` to `pyproject.toml` for correct PyPI rendering.\n- **`README-pypi.md`** — generated from `README.md` by `scripts/prepare_pypi_readme.py`; strips `pypi-strip` blocks and rewrites relative links to absolute GitHub URLs.\n- **Branch protection compatibility** — removed post-merge `build-plugin` CI job that pushed compiled plugin directly to `main`. Plugin sync is now enforced in PRs: CI fails if `synthadoc/data/obsidian-plugin/` is out of sync.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 5). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n"}, "8d13e736c1e0e0a28910d45a1d2f6808f46f2366": {"t": 1785767272.937619, "n": 190464, "h": "9f5d65725738a35c", "c": "# Synthadoc — Design Document\n\n**Version:** 1.0.1 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n29. [Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer)\n30. [Per-Source Truncation Flag](#30-per-source-truncation-flag)\n31. [Proportional Context Budget](#31-proportional-context-budget)\n32. [Knowledge Graph](#32-knowledge-graph)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent, checks CostGuard\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the full source text (bounded by `max_source_chars`) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `flag` / `skip`). Pages with `status='active'` are protected — sources that conflict with them trigger `flag` rather than `update` (RULE 1b).\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 0b — Key Data Extraction | None (regex) | Deterministic pre-processor: extracts numbers, percentages, currency amounts, ratios, formulas, underscore identifiers, and date ranges from source text via regex; appends them as a `[Key Data — extracted by pre-processor]` section before any LLM sees the text. Zero token cost; cannot hallucinate. Anchors the synthesis LLM to all quantitative candidates. |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads the full source text (bounded by `max_source_chars`) + BM25 candidates + `purpose.md` scope + status of each candidate page. Outputs per-page action: `create`, `update`, `flag`, `skip`. **RULE 1b:** a page with `status='active'` is human-reviewed and authoritative — when the source provides a conflicting value, date, formula, or conclusion, the action must be `flag`, not `update`. The only exception is a source that adds a topic the page does not yet mention at all. Decision cache version: `v2`. |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. If no citation markers are present in the returned body, a `citation_pass4_no_markers` WARNING audit event is written — the page is still saved but the condition is flagged for re-ingest with a more capable model. Zero-citation results are not cached so a subsequent re-ingest can succeed. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic. **Suppressed when TF fallback was used** (a small corpus with broad coverage may legitimately return fewer than 3 candidates — that is expected, not a gap).\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n| Citation presence (Check 5b) | Page body has ≥ 50 words and zero `^[filename:L-L]` citation markers → WARNING. Diagnostic only — does not block promotion to `active`. Indicates the annotation pass failed to produce markers, usually a model-compatibility issue (use Gemini 2.5 Flash or higher for reliable citation annotation). |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n**Compound identifier expansion:** When a token contains underscores, the tokenizer emits both the compound form and each component as separate tokens (`capex_growth` → `[\"capex_growth\", \"capex\", \"growth\"]`). This expansion is applied at both index time and query time so that human-friendly queries match programmatic identifiers and vice versa. Leading/trailing underscores are stripped before splitting; no empty strings are emitted.\n\n**TF fallback for small corpora:** When all BM25 scores in a result set are zero or negative (IDF collapse on a corpus of ≤ 2 documents, or a term present in every page), the search system falls back to normalised term frequency (TF) scoring: `TF(doc, query) = Σcount(term, doc) / len(doc_tokens)`. Results are returned in the same format as BM25 results with a `tf_fallback: True` flag on each `SearchResult`. This field is recorded in the audit trail and consumed by gap detection (see below).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp}` |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n| `GET` | `/graph` _(v1.0.0)_ | — | `{status, node_count, edge_count, cluster_count, nodes: [...], edges: [...]}` or `{status: \"computing\"}` on first call |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 0.6.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n**Reading View default:** `synthadoc plugin install` and `synthadoc plugin upgrade` write `\"defaultViewMode\": \"preview\"` to `.obsidian/app.json` after copying the plugin files. This causes Obsidian to open new notes in Reading View by default, which is required for `^[filename:L-L]` citation chips to render. Only `defaultViewMode` is written; all other keys in `app.json` are preserved. If `app.json` is malformed JSON the write is skipped and a warning is logged — the install still succeeds.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports dangling slugs — pages listed in ROUTING.md that no longer exist in the wiki (enabled only when ROUTING.md exists). **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [--wiki-root ] — generate ROUTING.md from index.md branch structure\n│ ├── validate [--wiki-root ] — report dangling slugs and cross-branch duplicates\n│ └── clean [--wiki-root ] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [--wiki-root ]\n├── candidates\n│ ├── list [--wiki-root ]\n│ ├── promote |--all [--wiki-root ]\n│ └── discard |--all [--wiki-root ]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Log warning, continue _(configured but not yet enforced — cost_guard is wired to the config but check() is not called in the ingest path)_ |\n| `cost.hard_gate_usd` | float | `2.00` | Require explicit confirmation _(configured but not yet enforced — see above)_ |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × backup_count` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n| `ingest.max_source_chars` | int | `32000` | Character limit applied to each source before the LLM call. Sources exceeding this limit are truncated; the page's `sources:` frontmatter entry gets `truncated: true` and lint emits a warning. Override per-run with `--max-source-chars N`. _(v1.0.0)_ |\n| `query.context_wiki_pct` | float | `0.60` | Fraction of the model context window reserved for wiki source pages. _(v1.0.0)_ |\n| `query.context_history_pct` | float | `0.20` | Fraction reserved for conversation history. _(v1.0.0)_ |\n| `query.context_system_pct` | float | `0.15` | Fraction reserved for system prompt and instructions. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_index_pct` | float | `0.05` | Fraction reserved for the wiki index preamble. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_window` | int | _(auto)_ | Override the built-in context window lookup for the configured model. Use when running a local or custom model whose window size is not in the built-in table. _(v1.0.0)_ |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python scripts/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python scripts/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement (soft warn / hard gate) is implemented and configurable but not yet wired into the LLM call path — it is infrastructure ready to activate.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | Log warning; auto-continue |\n| `hard_gate_usd` | $2.00 | Prompt user `Proceed? [y/N]`; block if N; skip prompt if `auto_confirm=True` or `--yes` flag |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` at OS socket level. Not configurable. No remote access without a separate reverse proxy (which the user must explicitly set up).\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `--wiki-root `.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; a missing source file triggers `archived`. Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: if the source path recorded in frontmatter no longer exists on disk, the page is transitioned to `archived`.\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 10) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 10 # turns to include in each request (default: 10; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp}` or `{error}` | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── hooks/ ← user hook scripts (always included)\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ ├── extracted/ ← text sidecars + PDF pagemaps (always included)\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `logs/`.\n\n> **v0.9.3 fix:** `hooks/` and `.synthadoc/extracted/` were previously missing from backups. Omitting `extracted/` caused the Source Viewer and Provenance citation modal to show \"Could not read source file\" after restore.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## 29. Pre-LLM Source Sanitizer\n\nEvery source document passes through a sanitizer immediately after text extraction, before any LLM call. The sanitizer removes six categories of content that could manipulate the LLM or degrade compilation quality:\n\n| Category | Action | Warning logged? |\n|---|---|---|\n| Zero-width characters (U+200B, U+200C, U+200D, U+FEFF) | Removed silently | No |\n| Bidi override characters (U+202A–U+202E, U+2066–U+2069) | Removed | Yes |\n| HTML comments (``) | Removed silently | No |\n| Hidden CSS spans (`display:none`, `visibility:hidden`) | Removed silently | No |\n| Base64 blobs ≥ 200 consecutive characters | Replaced with `[base64 content removed]` | Yes |\n| Instruction-override phrases (8 patterns: \"ignore previous instructions\", \"disregard the above\", \"override your system prompt\", etc.) | Replaced with `[redacted]` | Always |\n\nWhen content is removed and a warning is appropriate, the server logs a `WARN` entry:\n```\n[WARN] sanitizer stripped content from 'papers/survey.pdf': bidi overrides, instruction-override phrase\n```\n\nThis is a pure pre-processing step with zero LLM cost. Truncation (if configured) is applied after sanitization, so the character budget is consumed by clean text only.\n\n---\n\n## 30. Per-Source Truncation Flag\n\nLarge source documents are truncated before they reach the LLM. By default the limit is **32,000 characters** (~8,000 tokens), raised from the prior hardcoded 8,000-character limit. When a source exceeds this limit, its compiled wiki page records a `truncated: true` flag in its `sources:` frontmatter entry.\n\n### Configuration\n\n```toml\n[ingest]\nmax_source_chars = 32000 # default; raise for large PDFs or books\n```\n\n### Per-ingest override\n\nThe limit can be raised for a single ingest run without modifying the config:\n\n```bash\nsynthadoc ingest papers/large-textbook.pdf --max-source-chars 128000\n```\n\nThe same `max_source_chars` field is accepted by the HTTP `POST /jobs/ingest` body and the MCP `synthadoc_ingest` tool.\n\n### Lint surfacing\n\n`synthadoc lint` emits a warning for any page that has a truncated source:\n\n```\n[WARN] quantum-computing.md: source 'papers/large-survey.pdf' was truncated at ingest\n (source exceeded max_source_chars=32000 — 87,412 chars in source).\n To re-ingest with a higher limit (this source only):\n synthadoc ingest papers/large-survey.pdf --max-source-chars 128000\n To raise the limit for all future ingests:\n set [ingest] max_source_chars = 128000 in your config\n```\n\n---\n\n## 31. Proportional Context Budget\n\nQuery context is allocated proportionally to the configured model's context window, replacing the prior hard cap. The default allocation:\n\n| Slice | Default | Purpose |\n|---|---|---|\n| `context_wiki_pct` | 60% | Wiki page content (ranked by BM25 + vector score) |\n| `context_history_pct` | 20% | Chat turn history (newest-first, oldest dropped when over budget) |\n| `context_system_pct` | 15% | System prompt and purpose document |\n| `context_index_pct` | 5% | `ROUTING.md` and search index |\n\nPercentages must sum to ≤ 100. Configure in `[query]`:\n\n```toml\n[query]\ncontext_window = 0 # 0 = auto-detect from known model map\ncontext_wiki_pct = 60\ncontext_history_pct = 20\ncontext_system_pct = 15\ncontext_index_pct = 5\n```\n\n> **Note:** `context_system_pct` and `context_index_pct` are parsed and validated but not yet enforced as hard caps in this release — the system prompt uses a conservative fixed limit, which is well within the configured 15% slice for all supported models. Full per-slice enforcement is planned for v1.1.\n\n### Model context window map\n\nSynthadoc ships a built-in prefix-matched table:\n\n| Model | Context window |\n|---|---|\n| `claude-opus-4*`, `claude-sonnet-4*`, `claude-haiku-4*` | 200,000 tokens |\n| `gpt-4o*`, `gpt-4-turbo*` | 128,000 tokens |\n| `gpt-4` (exact) | 8,192 tokens |\n| `gpt-3.5-turbo*` | 16,385 tokens |\n| Unknown / fallback | 128,000 tokens |\n\nSet `context_window = N` in config to override the map (useful for local Ollama models with non-standard context sizes).\n\nAt the defaults on a 200k-token model: wiki slice ≈ 480,000 chars — far more than any practical query needs. On constrained models, the budget scales down automatically so the same config works across model families.\n\n---\n\n## 32. Knowledge Graph\n\nSynthadoc computes a wikilink graph across all active and draft pages during every lint run. The graph is stored in two tables in `audit.db`:\n\n```sql\nCREATE TABLE graph_nodes (\n slug TEXT PRIMARY KEY,\n cluster_id INTEGER NOT NULL,\n updated_at TEXT NOT NULL\n);\n\nCREATE TABLE graph_edges (\n from_slug TEXT NOT NULL,\n to_slug TEXT NOT NULL,\n weight INTEGER NOT NULL DEFAULT 1,\n PRIMARY KEY (from_slug, to_slug)\n);\n```\n\nEdge weight equals the number of `[[wikilink]]` references from one page to another (most pairs: 1). Cluster IDs are assigned by Louvain community detection — pages that link densely to each other end up in the same cluster.\n\n### REST API\n\n**`GET /graph`** — returns the full graph with per-node enrichment:\n\n```json\n{\n \"status\": \"ready\",\n \"node_count\": 45,\n \"edge_count\": 123,\n \"cluster_count\": 4,\n \"nodes\": [\n {\"slug\": \"quantum-error-correction\", \"title\": \"Quantum Error Correction\",\n \"type\": \"concept\", \"state\": \"active\", \"cluster_id\": 2}\n ],\n \"edges\": [\n {\"from\": \"quantum-error-correction\", \"to\": \"shor-algorithm\", \"weight\": 1}\n ]\n}\n```\n\nWhen the graph tables are empty (first call after upgrade, or before the first lint run), the server returns `{\"status\": \"computing\"}` and starts a background build immediately. The web UI polls every 2 seconds and renders the graph when the status becomes `\"ready\"`.\n\n### Web UI\n\nThe **Graph** tab (alongside Chat in the top nav) renders a D3.js force-directed graph. Nodes are colored by cluster. Clicking a node opens a sidebar showing the page title, type and lifecycle state badges, cluster assignment, and an **\"Ask about this →\"** button that switches to the Chat tab with a pre-filled query.\n\nControls: zoom in/out (scroll or pinch), drag nodes, filter by page type.\n\n---\n\n## Customization\n\nSynthadoc exposes four extension points — all hot-loaded, no server restart required.\n\n### Custom skills (new file formats)\n\nSubclass `BaseSkill` (Apache-2.0 — no AGPL obligation on your skill code), drop the folder in `/skills/` or `~/.synthadoc/skills/`, and Synthadoc picks it up on the next ingest. Skills match by file extension or intent prefix and support any Unicode text, including CJK prefixes.\n\n→ Full interface, manifest format, and examples: [§5 Skills System](#5-skills-system) and [§17 Plugin Development Guide](#17-plugin-development-guide)\n\n### Custom LLM providers\n\nSubclass `LLMProvider` from `synthadoc/providers/base.py` (Apache-2.0) and place the file in `~/.synthadoc/providers/` or the wiki `providers/` directory. Switch active provider with one config line.\n\n→ Full interface and wiring instructions: [§10 Configuration — Provider switching](#10-configuration)\n\n### Hooks\n\nShell scripts (any language) that fire on `on_ingest_complete` and `on_lint_complete`. Receive a JSON context on stdin. Set `blocking = true` to gate the operation on the hook's exit code — useful for CI pipelines and quality gates.\n\n→ Full event schema, context JSON, and blocking behaviour: [§11 Hook System](#11-hook-system)\n\n### Cache control\n\nThree cache layers (embedding, LLM response, provider prompt cache) invalidate automatically on source-file change (SHA-256). Force a fresh call with `--force`, or wipe all cached responses with `synthadoc cache clear -w `.\n\n→ Layer-by-layer breakdown and invalidation rules: [§12 Cache System](#12-cache-system)\n\n### Per-wiki AGENTS.md\n\nEdit `/AGENTS.md` to give the LLM domain-specific instructions — terminology, page-naming conventions, what to cross-reference. This is the highest-priority instruction source for every agent run against this wiki; the `scaffold` command regenerates a starter version from current wiki state.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v1.0.1\n\n- **Scaffold JSON extraction hardened** — replaced the greedy regex used to extract the LLM's JSON response with a brace-balanced parser; eliminates false-positive matches on `[[wikilink]]` syntax that previously caused `json.loads()` failures and silent scaffold corruption\n- **Generated file validation** — scaffold now validates every LLM-generated file (index, purpose, AGENTS.md) for minimum length, required frontmatter, and `[[wikilink]]` presence before writing to disk; validation failure marks the job as failed with a structured error message rather than silently writing a malformed page\n- **ROUTING.md regeneration validation** — `routing init` and scheduled routing regeneration now validate the regenerated file for minimum line count and anchor presence before applying it; prevents a corrupt or truncated ROUTING.md from entering the wiki\n- **Scaffold failure reporting** — scaffold skip and failure reasons are now surfaced in the terminal output and job error field; previously silent failures (e.g. LLM timeout, JSON parse error) are now visible without tailing the JSON log\n- **Install-time scaffold removed** — `synthadoc install` no longer runs the LLM scaffold call during installation; scaffold is a separate step (`synthadoc scaffold`) that the user runs after setting their API key, avoiding confusing API-key-not-set errors at install time\n- **AquaFlow LLM evaluation** — 15-question M&A due-diligence benchmark across five models (MiniMax-Think M3, Claude Opus 4.8, Claude Sonnet 4.6, DeepSeek-R1, Qwen Plus); full report at `docs/example/aquaflow/evaluation/report/llm-query-benchmark.md`\n\n### v1.0.0\n\n- **Pre-LLM source sanitizer** — every source document passes through a sanitizer before any LLM call; removes six categories of potentially harmful content (zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars, instruction-override phrases); a `WARN` log entry is emitted when bidi overrides or instruction-override phrases are stripped; zero LLM cost; applied before truncation so the character budget is consumed by clean text only\n- **Per-source truncation flag** — default character limit raised from 8,000 to 32,000 characters per source (~8,000 tokens); truncated sources are flagged with `truncated: true` in the page's `sources:` frontmatter; `--max-source-chars N` CLI flag and matching `POST /jobs/ingest` body field and MCP `synthadoc_ingest` parameter override the limit per run; `synthadoc lint` emits a warning for any page with a truncated source and suggests the override command; configurable via `[ingest] max_source_chars` in `config.toml`\n- **Proportional context budget** — query context is allocated proportionally to the configured model's context window; four configurable slices: `context_wiki_pct` (60%), `context_history_pct` (20%), `context_system_pct` (15%), `context_index_pct` (5%); built-in prefix-matched model context window table covers Claude 4, GPT-4o/turbo/3.5, and GPT-4; unknown models fall back to 128,000 tokens; `context_window = N` in `[query]` overrides the table for local or custom models\n- **Knowledge graph** — wikilink graph computed across all active and draft pages during every lint run; stored in `graph_nodes` and `graph_edges` tables in `audit.db`; edge weight = number of `[[wikilink]]` references between two pages; cluster IDs assigned by Louvain community detection; `GET /graph` REST endpoint returns nodes with per-node enrichment (title, type, state, cluster) and edges; first call after upgrade triggers a background build and returns `{\"status\":\"computing\"}`; web UI **Graph** tab (D3.js force-directed graph) — nodes colored by cluster, click to see page details and an \"Ask about this →\" button that opens a pre-filled chat query\n- **Citation accuracy & rendering improvements** — six correctness fixes to the ingest citation pipeline: (1) full source text passed to the decision LLM so citations can reference any line in the document; (2) `_annotate_citations` bugs A–F resolved (false-positive matches, off-by-one line ranges, duplicate markers, escaped bracket handling, multi-paragraph span errors, and source filename normalisation); (3) `_has_citations` check added — lint warns and suggests a model upgrade when a page has zero citation markers after synthesis; (4) `Key Data` section extraction hardened so numerical facts, rates, and formulas are preserved verbatim across the annotation pass; (5) `obsidianCitationsToGfm` rendering fixed in the web UI — citation superscripts now display correctly in the chat panel alongside streamed answers; (6) reading view set as the default Obsidian display mode for wiki pages\n\n### v0.9.3\n\n- **Backup now includes `hooks/` and `.synthadoc/extracted/`** — both directories were silently omitted from backups. `hooks/` contains user-customised hook scripts; `.synthadoc/extracted/` contains per-source text sidecars and PDF pagemaps written by the ingest agent. Without these, the Source Viewer and Provenance citation modal showed \"Could not read source file\" after restore.\n- **PyPI publish workflow** — `publish.yml` triggers on GitHub Release, builds the wheel with hatchling, and uploads to PyPI via OIDC trusted publishing (no API token stored). `pip install synthadoc` now works.\n- **Web UI bundled in pip package** — `synthadoc/data/web-ui/dist/` is now included in the wheel. `synthadoc web` previously returned 503 on pip-installed instances because the React dist was not packaged.\n- **CI sync checks** — PRs now fail if `synthadoc/data/obsidian-plugin/`, `synthadoc/data/web-ui/`, or `README-pypi.md` are out of sync with their source. All sync-check steps use `shell: bash` for Windows runner compatibility.\n- **`bump_version.py` extended** — now also updates `synthadoc/data/obsidian-plugin/manifest.json` alongside the source manifest, preventing version drift between the bundled copy and the plugin source.\n\n### v0.9.2\n\n- **`synthadoc/data/web-ui/dist/` packaged** — web UI dist files committed to the repo and bundled in the wheel via hatchling artifacts.\n- **`scripts/sync_web_ui.py`** — helper script to sync `web-ui/dist/` into `synthadoc/data/web-ui/dist/` after `npm run build`.\n\n### v0.9.1\n\n- **PyPI metadata** — added `readme`, `keywords`, `classifiers`, and `[project.urls]` to `pyproject.toml` for correct PyPI rendering.\n- **`README-pypi.md`** — generated from `README.md` by `scripts/prepare_pypi_readme.py`; strips `pypi-strip` blocks and rewrites relative links to absolute GitHub URLs.\n- **Branch protection compatibility** — removed post-merge `build-plugin` CI job that pushed compiled plugin directly to `main`. Plugin sync is now enforced in PRs: CI fails if `synthadoc/data/obsidian-plugin/` is out of sync.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 10). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n"}, "64cba0d7a83da4e7ffac7682ca88164f3b965a64": {"t": 1785767274.8255928, "n": 188832, "h": "41d322db7e37dcb5", "c": "# Synthadoc — Design Document\n\n**Version:** 1.0.0 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n29. [Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer)\n30. [Per-Source Truncation Flag](#30-per-source-truncation-flag)\n31. [Proportional Context Budget](#31-proportional-context-budget)\n32. [Knowledge Graph](#32-knowledge-graph)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent, checks CostGuard\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the full source text (bounded by `max_source_chars`) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `flag` / `skip`). Pages with `status='active'` are protected — sources that conflict with them trigger `flag` rather than `update` (RULE 1b).\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 0b — Key Data Extraction | None (regex) | Deterministic pre-processor: extracts numbers, percentages, currency amounts, ratios, formulas, underscore identifiers, and date ranges from source text via regex; appends them as a `[Key Data — extracted by pre-processor]` section before any LLM sees the text. Zero token cost; cannot hallucinate. Anchors the synthesis LLM to all quantitative candidates. |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads the full source text (bounded by `max_source_chars`) + BM25 candidates + `purpose.md` scope + status of each candidate page. Outputs per-page action: `create`, `update`, `flag`, `skip`. **RULE 1b:** a page with `status='active'` is human-reviewed and authoritative — when the source provides a conflicting value, date, formula, or conclusion, the action must be `flag`, not `update`. The only exception is a source that adds a topic the page does not yet mention at all. Decision cache version: `v2`. |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. If no citation markers are present in the returned body, a `citation_pass4_no_markers` WARNING audit event is written — the page is still saved but the condition is flagged for re-ingest with a more capable model. Zero-citation results are not cached so a subsequent re-ingest can succeed. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic. **Suppressed when TF fallback was used** (a small corpus with broad coverage may legitimately return fewer than 3 candidates — that is expected, not a gap).\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n| Citation presence (Check 5b) | Page body has ≥ 50 words and zero `^[filename:L-L]` citation markers → WARNING. Diagnostic only — does not block promotion to `active`. Indicates the annotation pass failed to produce markers, usually a model-compatibility issue (use Gemini 2.5 Flash or higher for reliable citation annotation). |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n**Compound identifier expansion:** When a token contains underscores, the tokenizer emits both the compound form and each component as separate tokens (`capex_growth` → `[\"capex_growth\", \"capex\", \"growth\"]`). This expansion is applied at both index time and query time so that human-friendly queries match programmatic identifiers and vice versa. Leading/trailing underscores are stripped before splitting; no empty strings are emitted.\n\n**TF fallback for small corpora:** When all BM25 scores in a result set are zero or negative (IDF collapse on a corpus of ≤ 2 documents, or a term present in every page), the search system falls back to normalised term frequency (TF) scoring: `TF(doc, query) = Σcount(term, doc) / len(doc_tokens)`. Results are returned in the same format as BM25 results with a `tf_fallback: True` flag on each `SearchResult`. This field is recorded in the audit trail and consumed by gap detection (see below).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp}` |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n| `GET` | `/graph` _(v1.0.0)_ | — | `{status, node_count, edge_count, cluster_count, nodes: [...], edges: [...]}` or `{status: \"computing\"}` on first call |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 0.6.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n**Reading View default:** `synthadoc plugin install` and `synthadoc plugin upgrade` write `\"defaultViewMode\": \"preview\"` to `.obsidian/app.json` after copying the plugin files. This causes Obsidian to open new notes in Reading View by default, which is required for `^[filename:L-L]` citation chips to render. Only `defaultViewMode` is written; all other keys in `app.json` are preserved. If `app.json` is malformed JSON the write is skipped and a warning is logged — the install still succeeds.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports dangling slugs — pages listed in ROUTING.md that no longer exist in the wiki (enabled only when ROUTING.md exists). **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [--wiki-root ] — generate ROUTING.md from index.md branch structure\n│ ├── validate [--wiki-root ] — report dangling slugs and cross-branch duplicates\n│ └── clean [--wiki-root ] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [--wiki-root ]\n├── candidates\n│ ├── list [--wiki-root ]\n│ ├── promote |--all [--wiki-root ]\n│ └── discard |--all [--wiki-root ]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Log warning, continue _(configured but not yet enforced — cost_guard is wired to the config but check() is not called in the ingest path)_ |\n| `cost.hard_gate_usd` | float | `2.00` | Require explicit confirmation _(configured but not yet enforced — see above)_ |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × backup_count` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n| `ingest.max_source_chars` | int | `32000` | Character limit applied to each source before the LLM call. Sources exceeding this limit are truncated; the page's `sources:` frontmatter entry gets `truncated: true` and lint emits a warning. Override per-run with `--max-source-chars N`. _(v1.0.0)_ |\n| `query.context_wiki_pct` | float | `0.60` | Fraction of the model context window reserved for wiki source pages. _(v1.0.0)_ |\n| `query.context_history_pct` | float | `0.20` | Fraction reserved for conversation history. _(v1.0.0)_ |\n| `query.context_system_pct` | float | `0.15` | Fraction reserved for system prompt and instructions. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_index_pct` | float | `0.05` | Fraction reserved for the wiki index preamble. Parsed and validated but not yet enforced as a hard cap in v1.0.0. _(v1.0.0)_ |\n| `query.context_window` | int | _(auto)_ | Override the built-in context window lookup for the configured model. Use when running a local or custom model whose window size is not in the built-in table. _(v1.0.0)_ |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python scripts/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python scripts/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement (soft warn / hard gate) is implemented and configurable but not yet wired into the LLM call path — it is infrastructure ready to activate.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | Log warning; auto-continue |\n| `hard_gate_usd` | $2.00 | Prompt user `Proceed? [y/N]`; block if N; skip prompt if `auto_confirm=True` or `--yes` flag |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` at OS socket level. Not configurable. No remote access without a separate reverse proxy (which the user must explicitly set up).\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `--wiki-root `.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; a missing source file triggers `archived`. Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: if the source path recorded in frontmatter no longer exists on disk, the page is transitioned to `archived`.\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 10) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 10 # turns to include in each request (default: 10; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp}` or `{error}` | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── hooks/ ← user hook scripts (always included)\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ ├── extracted/ ← text sidecars + PDF pagemaps (always included)\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `logs/`.\n\n> **v0.9.3 fix:** `hooks/` and `.synthadoc/extracted/` were previously missing from backups. Omitting `extracted/` caused the Source Viewer and Provenance citation modal to show \"Could not read source file\" after restore.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## 29. Pre-LLM Source Sanitizer\n\nEvery source document passes through a sanitizer immediately after text extraction, before any LLM call. The sanitizer removes six categories of content that could manipulate the LLM or degrade compilation quality:\n\n| Category | Action | Warning logged? |\n|---|---|---|\n| Zero-width characters (U+200B, U+200C, U+200D, U+FEFF) | Removed silently | No |\n| Bidi override characters (U+202A–U+202E, U+2066–U+2069) | Removed | Yes |\n| HTML comments (``) | Removed silently | No |\n| Hidden CSS spans (`display:none`, `visibility:hidden`) | Removed silently | No |\n| Base64 blobs ≥ 200 consecutive characters | Replaced with `[base64 content removed]` | Yes |\n| Instruction-override phrases (8 patterns: \"ignore previous instructions\", \"disregard the above\", \"override your system prompt\", etc.) | Replaced with `[redacted]` | Always |\n\nWhen content is removed and a warning is appropriate, the server logs a `WARN` entry:\n```\n[WARN] sanitizer stripped content from 'papers/survey.pdf': bidi overrides, instruction-override phrase\n```\n\nThis is a pure pre-processing step with zero LLM cost. Truncation (if configured) is applied after sanitization, so the character budget is consumed by clean text only.\n\n---\n\n## 30. Per-Source Truncation Flag\n\nLarge source documents are truncated before they reach the LLM. By default the limit is **32,000 characters** (~8,000 tokens), raised from the prior hardcoded 8,000-character limit. When a source exceeds this limit, its compiled wiki page records a `truncated: true` flag in its `sources:` frontmatter entry.\n\n### Configuration\n\n```toml\n[ingest]\nmax_source_chars = 32000 # default; raise for large PDFs or books\n```\n\n### Per-ingest override\n\nThe limit can be raised for a single ingest run without modifying the config:\n\n```bash\nsynthadoc ingest papers/large-textbook.pdf --max-source-chars 128000\n```\n\nThe same `max_source_chars` field is accepted by the HTTP `POST /jobs/ingest` body and the MCP `synthadoc_ingest` tool.\n\n### Lint surfacing\n\n`synthadoc lint` emits a warning for any page that has a truncated source:\n\n```\n[WARN] quantum-computing.md: source 'papers/large-survey.pdf' was truncated at ingest\n (source exceeded max_source_chars=32000 — 87,412 chars in source).\n To re-ingest with a higher limit (this source only):\n synthadoc ingest papers/large-survey.pdf --max-source-chars 128000\n To raise the limit for all future ingests:\n set [ingest] max_source_chars = 128000 in your config\n```\n\n---\n\n## 31. Proportional Context Budget\n\nQuery context is allocated proportionally to the configured model's context window, replacing the prior hard cap. The default allocation:\n\n| Slice | Default | Purpose |\n|---|---|---|\n| `context_wiki_pct` | 60% | Wiki page content (ranked by BM25 + vector score) |\n| `context_history_pct` | 20% | Chat turn history (newest-first, oldest dropped when over budget) |\n| `context_system_pct` | 15% | System prompt and purpose document |\n| `context_index_pct` | 5% | `ROUTING.md` and search index |\n\nPercentages must sum to ≤ 100. Configure in `[query]`:\n\n```toml\n[query]\ncontext_window = 0 # 0 = auto-detect from known model map\ncontext_wiki_pct = 60\ncontext_history_pct = 20\ncontext_system_pct = 15\ncontext_index_pct = 5\n```\n\n> **Note:** `context_system_pct` and `context_index_pct` are parsed and validated but not yet enforced as hard caps in this release — the system prompt uses a conservative fixed limit, which is well within the configured 15% slice for all supported models. Full per-slice enforcement is planned for v1.1.\n\n### Model context window map\n\nSynthadoc ships a built-in prefix-matched table:\n\n| Model | Context window |\n|---|---|\n| `claude-opus-4*`, `claude-sonnet-4*`, `claude-haiku-4*` | 200,000 tokens |\n| `gpt-4o*`, `gpt-4-turbo*` | 128,000 tokens |\n| `gpt-4` (exact) | 8,192 tokens |\n| `gpt-3.5-turbo*` | 16,385 tokens |\n| Unknown / fallback | 128,000 tokens |\n\nSet `context_window = N` in config to override the map (useful for local Ollama models with non-standard context sizes).\n\nAt the defaults on a 200k-token model: wiki slice ≈ 480,000 chars — far more than any practical query needs. On constrained models, the budget scales down automatically so the same config works across model families.\n\n---\n\n## 32. Knowledge Graph\n\nSynthadoc computes a wikilink graph across all active and draft pages during every lint run. The graph is stored in two tables in `audit.db`:\n\n```sql\nCREATE TABLE graph_nodes (\n slug TEXT PRIMARY KEY,\n cluster_id INTEGER NOT NULL,\n updated_at TEXT NOT NULL\n);\n\nCREATE TABLE graph_edges (\n from_slug TEXT NOT NULL,\n to_slug TEXT NOT NULL,\n weight INTEGER NOT NULL DEFAULT 1,\n PRIMARY KEY (from_slug, to_slug)\n);\n```\n\nEdge weight equals the number of `[[wikilink]]` references from one page to another (most pairs: 1). Cluster IDs are assigned by Louvain community detection — pages that link densely to each other end up in the same cluster.\n\n### REST API\n\n**`GET /graph`** — returns the full graph with per-node enrichment:\n\n```json\n{\n \"status\": \"ready\",\n \"node_count\": 45,\n \"edge_count\": 123,\n \"cluster_count\": 4,\n \"nodes\": [\n {\"slug\": \"quantum-error-correction\", \"title\": \"Quantum Error Correction\",\n \"type\": \"concept\", \"state\": \"active\", \"cluster_id\": 2}\n ],\n \"edges\": [\n {\"from\": \"quantum-error-correction\", \"to\": \"shor-algorithm\", \"weight\": 1}\n ]\n}\n```\n\nWhen the graph tables are empty (first call after upgrade, or before the first lint run), the server returns `{\"status\": \"computing\"}` and starts a background build immediately. The web UI polls every 2 seconds and renders the graph when the status becomes `\"ready\"`.\n\n### Web UI\n\nThe **Graph** tab (alongside Chat in the top nav) renders a D3.js force-directed graph. Nodes are colored by cluster. Clicking a node opens a sidebar showing the page title, type and lifecycle state badges, cluster assignment, and an **\"Ask about this →\"** button that switches to the Chat tab with a pre-filled query.\n\nControls: zoom in/out (scroll or pinch), drag nodes, filter by page type.\n\n---\n\n## Customization\n\nSynthadoc exposes four extension points — all hot-loaded, no server restart required.\n\n### Custom skills (new file formats)\n\nSubclass `BaseSkill` (Apache-2.0 — no AGPL obligation on your skill code), drop the folder in `/skills/` or `~/.synthadoc/skills/`, and Synthadoc picks it up on the next ingest. Skills match by file extension or intent prefix and support any Unicode text, including CJK prefixes.\n\n→ Full interface, manifest format, and examples: [§5 Skills System](#5-skills-system) and [§17 Plugin Development Guide](#17-plugin-development-guide)\n\n### Custom LLM providers\n\nSubclass `LLMProvider` from `synthadoc/providers/base.py` (Apache-2.0) and place the file in `~/.synthadoc/providers/` or the wiki `providers/` directory. Switch active provider with one config line.\n\n→ Full interface and wiring instructions: [§10 Configuration — Provider switching](#10-configuration)\n\n### Hooks\n\nShell scripts (any language) that fire on `on_ingest_complete` and `on_lint_complete`. Receive a JSON context on stdin. Set `blocking = true` to gate the operation on the hook's exit code — useful for CI pipelines and quality gates.\n\n→ Full event schema, context JSON, and blocking behaviour: [§11 Hook System](#11-hook-system)\n\n### Cache control\n\nThree cache layers (embedding, LLM response, provider prompt cache) invalidate automatically on source-file change (SHA-256). Force a fresh call with `--force`, or wipe all cached responses with `synthadoc cache clear -w `.\n\n→ Layer-by-layer breakdown and invalidation rules: [§12 Cache System](#12-cache-system)\n\n### Per-wiki AGENTS.md\n\nEdit `/AGENTS.md` to give the LLM domain-specific instructions — terminology, page-naming conventions, what to cross-reference. This is the highest-priority instruction source for every agent run against this wiki; the `scaffold` command regenerates a starter version from current wiki state.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v1.0.0\n\n- **Pre-LLM source sanitizer** — every source document passes through a sanitizer before any LLM call; removes six categories of potentially harmful content (zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars, instruction-override phrases); a `WARN` log entry is emitted when bidi overrides or instruction-override phrases are stripped; zero LLM cost; applied before truncation so the character budget is consumed by clean text only\n- **Per-source truncation flag** — default character limit raised from 8,000 to 32,000 characters per source (~8,000 tokens); truncated sources are flagged with `truncated: true` in the page's `sources:` frontmatter; `--max-source-chars N` CLI flag and matching `POST /jobs/ingest` body field and MCP `synthadoc_ingest` parameter override the limit per run; `synthadoc lint` emits a warning for any page with a truncated source and suggests the override command; configurable via `[ingest] max_source_chars` in `config.toml`\n- **Proportional context budget** — query context is allocated proportionally to the configured model's context window; four configurable slices: `context_wiki_pct` (60%), `context_history_pct` (20%), `context_system_pct` (15%), `context_index_pct` (5%); built-in prefix-matched model context window table covers Claude 4, GPT-4o/turbo/3.5, and GPT-4; unknown models fall back to 128,000 tokens; `context_window = N` in `[query]` overrides the table for local or custom models\n- **Knowledge graph** — wikilink graph computed across all active and draft pages during every lint run; stored in `graph_nodes` and `graph_edges` tables in `audit.db`; edge weight = number of `[[wikilink]]` references between two pages; cluster IDs assigned by Louvain community detection; `GET /graph` REST endpoint returns nodes with per-node enrichment (title, type, state, cluster) and edges; first call after upgrade triggers a background build and returns `{\"status\":\"computing\"}`; web UI **Graph** tab (D3.js force-directed graph) — nodes colored by cluster, click to see page details and an \"Ask about this →\" button that opens a pre-filled chat query\n- **Citation accuracy & rendering improvements** — six correctness fixes to the ingest citation pipeline: (1) full source text passed to the decision LLM so citations can reference any line in the document; (2) `_annotate_citations` bugs A–F resolved (false-positive matches, off-by-one line ranges, duplicate markers, escaped bracket handling, multi-paragraph span errors, and source filename normalisation); (3) `_has_citations` check added — lint warns and suggests a model upgrade when a page has zero citation markers after synthesis; (4) `Key Data` section extraction hardened so numerical facts, rates, and formulas are preserved verbatim across the annotation pass; (5) `obsidianCitationsToGfm` rendering fixed in the web UI — citation superscripts now display correctly in the chat panel alongside streamed answers; (6) reading view set as the default Obsidian display mode for wiki pages\n\n### v0.9.3\n\n- **Backup now includes `hooks/` and `.synthadoc/extracted/`** — both directories were silently omitted from backups. `hooks/` contains user-customised hook scripts; `.synthadoc/extracted/` contains per-source text sidecars and PDF pagemaps written by the ingest agent. Without these, the Source Viewer and Provenance citation modal showed \"Could not read source file\" after restore.\n- **PyPI publish workflow** — `publish.yml` triggers on GitHub Release, builds the wheel with hatchling, and uploads to PyPI via OIDC trusted publishing (no API token stored). `pip install synthadoc` now works.\n- **Web UI bundled in pip package** — `synthadoc/data/web-ui/dist/` is now included in the wheel. `synthadoc web` previously returned 503 on pip-installed instances because the React dist was not packaged.\n- **CI sync checks** — PRs now fail if `synthadoc/data/obsidian-plugin/`, `synthadoc/data/web-ui/`, or `README-pypi.md` are out of sync with their source. All sync-check steps use `shell: bash` for Windows runner compatibility.\n- **`bump_version.py` extended** — now also updates `synthadoc/data/obsidian-plugin/manifest.json` alongside the source manifest, preventing version drift between the bundled copy and the plugin source.\n\n### v0.9.2\n\n- **`synthadoc/data/web-ui/dist/` packaged** — web UI dist files committed to the repo and bundled in the wheel via hatchling artifacts.\n- **`scripts/sync_web_ui.py`** — helper script to sync `web-ui/dist/` into `synthadoc/data/web-ui/dist/` after `npm run build`.\n\n### v0.9.1\n\n- **PyPI metadata** — added `readme`, `keywords`, `classifiers`, and `[project.urls]` to `pyproject.toml` for correct PyPI rendering.\n- **`README-pypi.md`** — generated from `README.md` by `scripts/prepare_pypi_readme.py`; strips `pypi-strip` blocks and rewrites relative links to absolute GitHub URLs.\n- **Branch protection compatibility** — removed post-merge `build-plugin` CI job that pushed compiled plugin directly to `main`. Plugin sync is now enforced in PRs: CI fails if `synthadoc/data/obsidian-plugin/` is out of sync.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 10). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n"}, "9ae82a65002c24030de75c738667b1fbd1d00785": {"t": 1785767276.6576009, "n": 183545, "h": "603d0e9b6df775ce", "c": "# Synthadoc — Design Document\n\n**Version:** 1.0.0 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n29. [Pre-LLM Source Sanitizer](#29-pre-llm-source-sanitizer)\n30. [Per-Source Truncation Flag](#30-per-source-truncation-flag)\n31. [Proportional Context Budget](#31-proportional-context-budget)\n32. [Knowledge Graph](#32-knowledge-graph)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent, checks CostGuard\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the summary (not raw text) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `skip` / `flag_contradiction`)\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads summary (not full text) + BM25 candidates + `purpose.md` scope. Outputs per-page action: `create`, `update`, `flag`, `skip` |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp}` |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 0.6.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports dangling slugs — pages listed in ROUTING.md that no longer exist in the wiki (enabled only when ROUTING.md exists). **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [--wiki-root ] — generate ROUTING.md from index.md branch structure\n│ ├── validate [--wiki-root ] — report dangling slugs and cross-branch duplicates\n│ └── clean [--wiki-root ] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [--wiki-root ]\n├── candidates\n│ ├── list [--wiki-root ]\n│ ├── promote |--all [--wiki-root ]\n│ └── discard |--all [--wiki-root ]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Log warning, continue _(configured but not yet enforced — cost_guard is wired to the config but check() is not called in the ingest path)_ |\n| `cost.hard_gate_usd` | float | `2.00` | Require explicit confirmation _(configured but not yet enforced — see above)_ |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × backup_count` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python scripts/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python scripts/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement (soft warn / hard gate) is implemented and configurable but not yet wired into the LLM call path — it is infrastructure ready to activate.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | Log warning; auto-continue |\n| `hard_gate_usd` | $2.00 | Prompt user `Proceed? [y/N]`; block if N; skip prompt if `auto_confirm=True` or `--yes` flag |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` at OS socket level. Not configurable. No remote access without a separate reverse proxy (which the user must explicitly set up).\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `--wiki-root `.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; a missing source file triggers `archived`. Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: if the source path recorded in frontmatter no longer exists on disk, the page is transitioned to `archived`.\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 10) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 10 # turns to include in each request (default: 10; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp}` or `{error}` | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── hooks/ ← user hook scripts (always included)\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ ├── extracted/ ← text sidecars + PDF pagemaps (always included)\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `logs/`.\n\n> **v0.9.3 fix:** `hooks/` and `.synthadoc/extracted/` were previously missing from backups. Omitting `extracted/` caused the Source Viewer and Provenance citation modal to show \"Could not read source file\" after restore.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## 29. Pre-LLM Source Sanitizer\n\nEvery source document passes through a sanitizer immediately after text extraction, before any LLM call. The sanitizer removes six categories of content that could manipulate the LLM or degrade compilation quality:\n\n| Category | Action | Warning logged? |\n|---|---|---|\n| Zero-width characters (U+200B, U+200C, U+200D, U+FEFF) | Removed silently | No |\n| Bidi override characters (U+202A–U+202E, U+2066–U+2069) | Removed | Yes |\n| HTML comments (``) | Removed silently | No |\n| Hidden CSS spans (`display:none`, `visibility:hidden`) | Removed silently | No |\n| Base64 blobs ≥ 200 consecutive characters | Replaced with `[base64 content removed]` | Yes |\n| Instruction-override phrases (8 patterns: \"ignore previous instructions\", \"disregard the above\", \"override your system prompt\", etc.) | Replaced with `[redacted]` | Always |\n\nWhen content is removed and a warning is appropriate, the server logs a `WARN` entry:\n```\n[WARN] sanitizer stripped content from 'papers/survey.pdf': bidi overrides, instruction-override phrase\n```\n\nThis is a pure pre-processing step with zero LLM cost. Truncation (if configured) is applied after sanitization, so the character budget is consumed by clean text only.\n\n---\n\n## 30. Per-Source Truncation Flag\n\nLarge source documents are truncated before they reach the LLM. By default the limit is **32,000 characters** (~8,000 tokens), raised from the prior hardcoded 8,000-character limit. When a source exceeds this limit, its compiled wiki page records a `truncated: true` flag in its `sources:` frontmatter entry.\n\n### Configuration\n\n```toml\n[ingest]\nmax_source_chars = 32000 # default; raise for large PDFs or books\n```\n\n### Per-ingest override\n\nThe limit can be raised for a single ingest run without modifying the config:\n\n```bash\nsynthadoc ingest papers/large-textbook.pdf --max-source-chars 128000\n```\n\nThe same `max_source_chars` field is accepted by the HTTP `POST /jobs/ingest` body and the MCP `synthadoc_ingest` tool.\n\n### Lint surfacing\n\n`synthadoc lint` emits a warning for any page that has a truncated source:\n\n```\n[WARN] quantum-computing.md: source 'papers/large-survey.pdf' was truncated at ingest\n (source exceeded max_source_chars=32000 — 87,412 chars in source).\n To re-ingest with a higher limit (this source only):\n synthadoc ingest papers/large-survey.pdf --max-source-chars 128000\n To raise the limit for all future ingests:\n set [ingest] max_source_chars = 128000 in your config\n```\n\n---\n\n## 31. Proportional Context Budget\n\nQuery context is allocated proportionally to the configured model's context window, replacing the prior hard cap. The default allocation:\n\n| Slice | Default | Purpose |\n|---|---|---|\n| `context_wiki_pct` | 60% | Wiki page content (ranked by BM25 + vector score) |\n| `context_history_pct` | 20% | Chat turn history (newest-first, oldest dropped when over budget) |\n| `context_system_pct` | 15% | System prompt and purpose document |\n| `context_index_pct` | 5% | `ROUTING.md` and search index |\n\nPercentages must sum to ≤ 100. Configure in `[query]`:\n\n```toml\n[query]\ncontext_window = 0 # 0 = auto-detect from known model map\ncontext_wiki_pct = 60\ncontext_history_pct = 20\ncontext_system_pct = 15\ncontext_index_pct = 5\n```\n\n> **Note:** `context_system_pct` and `context_index_pct` are parsed and validated but not yet enforced as hard caps in this release — the system prompt uses a conservative fixed limit, which is well within the configured 15% slice for all supported models. Full per-slice enforcement is planned for v1.1.\n\n### Model context window map\n\nSynthadoc ships a built-in prefix-matched table:\n\n| Model | Context window |\n|---|---|\n| `claude-opus-4*`, `claude-sonnet-4*`, `claude-haiku-4*` | 200,000 tokens |\n| `gpt-4o*`, `gpt-4-turbo*` | 128,000 tokens |\n| `gpt-4` (exact) | 8,192 tokens |\n| `gpt-3.5-turbo*` | 16,385 tokens |\n| Unknown / fallback | 128,000 tokens |\n\nSet `context_window = N` in config to override the map (useful for local Ollama models with non-standard context sizes).\n\nAt the defaults on a 200k-token model: wiki slice ≈ 480,000 chars — far more than any practical query needs. On constrained models, the budget scales down automatically so the same config works across model families.\n\n---\n\n## 32. Knowledge Graph\n\nSynthadoc computes a wikilink graph across all active and draft pages during every lint run. The graph is stored in two tables in `audit.db`:\n\n```sql\nCREATE TABLE graph_nodes (\n slug TEXT PRIMARY KEY,\n cluster_id INTEGER NOT NULL,\n updated_at TEXT NOT NULL\n);\n\nCREATE TABLE graph_edges (\n from_slug TEXT NOT NULL,\n to_slug TEXT NOT NULL,\n weight INTEGER NOT NULL DEFAULT 1,\n PRIMARY KEY (from_slug, to_slug)\n);\n```\n\nEdge weight equals the number of `[[wikilink]]` references from one page to another (most pairs: 1). Cluster IDs are assigned by Louvain community detection — pages that link densely to each other end up in the same cluster.\n\n### REST API\n\n**`GET /graph`** — returns the full graph with per-node enrichment:\n\n```json\n{\n \"status\": \"ready\",\n \"node_count\": 45,\n \"edge_count\": 123,\n \"cluster_count\": 4,\n \"nodes\": [\n {\"slug\": \"quantum-error-correction\", \"title\": \"Quantum Error Correction\",\n \"type\": \"concept\", \"state\": \"active\", \"cluster_id\": 2}\n ],\n \"edges\": [\n {\"from\": \"quantum-error-correction\", \"to\": \"shor-algorithm\", \"weight\": 1}\n ]\n}\n```\n\nWhen the graph tables are empty (first call after upgrade, or before the first lint run), the server returns `{\"status\": \"computing\"}` and starts a background build immediately. The web UI polls every 2 seconds and renders the graph when the status becomes `\"ready\"`.\n\n### Web UI\n\nThe **Graph** tab (alongside Chat in the top nav) renders a D3.js force-directed graph. Nodes are colored by cluster. Clicking a node opens a sidebar showing the page title, type and lifecycle state badges, cluster assignment, and an **\"Ask about this →\"** button that switches to the Chat tab with a pre-filled query.\n\nControls: zoom in/out (scroll or pinch), drag nodes, filter by page type.\n\n---\n\n## Customization\n\nSynthadoc exposes four extension points — all hot-loaded, no server restart required.\n\n### Custom skills (new file formats)\n\nSubclass `BaseSkill` (Apache-2.0 — no AGPL obligation on your skill code), drop the folder in `/skills/` or `~/.synthadoc/skills/`, and Synthadoc picks it up on the next ingest. Skills match by file extension or intent prefix and support any Unicode text, including CJK prefixes.\n\n→ Full interface, manifest format, and examples: [§5 Skills System](#5-skills-system) and [§17 Plugin Development Guide](#17-plugin-development-guide)\n\n### Custom LLM providers\n\nSubclass `LLMProvider` from `synthadoc/providers/base.py` (Apache-2.0) and place the file in `~/.synthadoc/providers/` or the wiki `providers/` directory. Switch active provider with one config line.\n\n→ Full interface and wiring instructions: [§10 Configuration — Provider switching](#10-configuration)\n\n### Hooks\n\nShell scripts (any language) that fire on `on_ingest_complete` and `on_lint_complete`. Receive a JSON context on stdin. Set `blocking = true` to gate the operation on the hook's exit code — useful for CI pipelines and quality gates.\n\n→ Full event schema, context JSON, and blocking behaviour: [§11 Hook System](#11-hook-system)\n\n### Cache control\n\nThree cache layers (embedding, LLM response, provider prompt cache) invalidate automatically on source-file change (SHA-256). Force a fresh call with `--force`, or wipe all cached responses with `synthadoc cache clear -w `.\n\n→ Layer-by-layer breakdown and invalidation rules: [§12 Cache System](#12-cache-system)\n\n### Per-wiki AGENTS.md\n\nEdit `/AGENTS.md` to give the LLM domain-specific instructions — terminology, page-naming conventions, what to cross-reference. This is the highest-priority instruction source for every agent run against this wiki; the `scaffold` command regenerates a starter version from current wiki state.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 10). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v1.0.0\n\n- **Pre-LLM source sanitizer** — every source document passes through a sanitizer before any LLM call; removes six categories of potentially harmful content (zero-width characters, bidi overrides, HTML comments, hidden CSS spans, base64 blobs ≥ 200 chars, instruction-override phrases); a `WARN` log entry is emitted when bidi overrides or instruction-override phrases are stripped; zero LLM cost; applied before truncation so the character budget is consumed by clean text only\n- **Per-source truncation flag** — default character limit raised from 8,000 to 32,000 characters per source (~8,000 tokens); truncated sources are flagged with `truncated: true` in the page's `sources:` frontmatter; `--max-source-chars N` CLI flag and matching `POST /jobs/ingest` body field and MCP `synthadoc_ingest` parameter override the limit per run; `synthadoc lint` emits a warning for any page with a truncated source and suggests the override command; configurable via `[ingest] max_source_chars` in `config.toml`\n- **Proportional context budget** — query context is allocated proportionally to the configured model's context window; four configurable slices: `context_wiki_pct` (60%), `context_history_pct` (20%), `context_system_pct` (15%), `context_index_pct` (5%); built-in prefix-matched model context window table covers Claude 4, GPT-4o/turbo/3.5, and GPT-4; unknown models fall back to 128,000 tokens; `context_window = N` in `[query]` overrides the table for local or custom models\n- **Knowledge graph** — wikilink graph computed across all active and draft pages during every lint run; stored in `graph_nodes` and `graph_edges` tables in `audit.db`; edge weight = number of `[[wikilink]]` references between two pages; cluster IDs assigned by Louvain community detection; `GET /graph` REST endpoint returns nodes with per-node enrichment (title, type, state, cluster) and edges; first call after upgrade triggers a background build and returns `{\"status\":\"computing\"}`; web UI **Graph** tab (D3.js force-directed graph) — nodes colored by cluster, click to see page details and an \"Ask about this →\" button that opens a pre-filled chat query\n\n### v0.9.3\n\n- **Backup now includes `hooks/` and `.synthadoc/extracted/`** — both directories were silently omitted from backups. `hooks/` contains user-customised hook scripts; `.synthadoc/extracted/` contains per-source text sidecars and PDF pagemaps written by the ingest agent. Without these, the Source Viewer and Provenance citation modal showed \"Could not read source file\" after restore.\n- **PyPI publish workflow** — `publish.yml` triggers on GitHub Release, builds the wheel with hatchling, and uploads to PyPI via OIDC trusted publishing (no API token stored). `pip install synthadoc` now works.\n- **Web UI bundled in pip package** — `synthadoc/data/web-ui/dist/` is now included in the wheel. `synthadoc web` previously returned 503 on pip-installed instances because the React dist was not packaged.\n- **CI sync checks** — PRs now fail if `synthadoc/data/obsidian-plugin/`, `synthadoc/data/web-ui/`, or `README-pypi.md` are out of sync with their source. All sync-check steps use `shell: bash` for Windows runner compatibility.\n- **`bump_version.py` extended** — now also updates `synthadoc/data/obsidian-plugin/manifest.json` alongside the source manifest, preventing version drift between the bundled copy and the plugin source.\n\n### v0.9.2\n\n- **`synthadoc/data/web-ui/dist/` packaged** — web UI dist files committed to the repo and bundled in the wheel via hatchling artifacts.\n- **`scripts/sync_web_ui.py`** — helper script to sync `web-ui/dist/` into `synthadoc/data/web-ui/dist/` after `npm run build`.\n\n### v0.9.1\n\n- **PyPI metadata** — added `readme`, `keywords`, `classifiers`, and `[project.urls]` to `pyproject.toml` for correct PyPI rendering.\n- **`README-pypi.md`** — generated from `README.md` by `scripts/prepare_pypi_readme.py`; strips `pypi-strip` blocks and rewrites relative links to absolute GitHub URLs.\n- **Branch protection compatibility** — removed post-merge `build-plugin` CI job that pushed compiled plugin directly to `main`. Plugin sync is now enforced in PRs: CI fails if `synthadoc/data/obsidian-plugin/` is out of sync.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n"}, "0d6caa1827e190c5a9c291890846ba73035e67f9": {"t": 1785767278.4024334, "n": 173086, "h": "0c47ab8f2290ecf7", "c": "# Synthadoc — Design Document\n\n**Version:** 0.9.3 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent, checks CostGuard\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the summary (not raw text) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `skip` / `flag_contradiction`)\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads summary (not full text) + BM25 candidates + `purpose.md` scope. Outputs per-page action: `create`, `update`, `flag`, `skip` |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp}` |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 0.6.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports dangling slugs — pages listed in ROUTING.md that no longer exist in the wiki (enabled only when ROUTING.md exists). **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [--wiki-root ] — generate ROUTING.md from index.md branch structure\n│ ├── validate [--wiki-root ] — report dangling slugs and cross-branch duplicates\n│ └── clean [--wiki-root ] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [--wiki-root ]\n├── candidates\n│ ├── list [--wiki-root ]\n│ ├── promote |--all [--wiki-root ]\n│ └── discard |--all [--wiki-root ]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Log warning, continue _(configured but not yet enforced — cost_guard is wired to the config but check() is not called in the ingest path)_ |\n| `cost.hard_gate_usd` | float | `2.00` | Require explicit confirmation _(configured but not yet enforced — see above)_ |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × backup_count` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python scripts/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python scripts/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement (soft warn / hard gate) is implemented and configurable but not yet wired into the LLM call path — it is infrastructure ready to activate.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | Log warning; auto-continue |\n| `hard_gate_usd` | $2.00 | Prompt user `Proceed? [y/N]`; block if N; skip prompt if `auto_confirm=True` or `--yes` flag |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` at OS socket level. Not configurable. No remote access without a separate reverse proxy (which the user must explicitly set up).\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `--wiki-root `.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; a missing source file triggers `archived`. Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: if the source path recorded in frontmatter no longer exists on disk, the page is transitioned to `archived`.\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 10) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 10 # turns to include in each request (default: 10; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp}` or `{error}` | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── hooks/ ← user hook scripts (always included)\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ ├── extracted/ ← text sidecars + PDF pagemaps (always included)\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `logs/`.\n\n> **v0.9.3 fix:** `hooks/` and `.synthadoc/extracted/` were previously missing from backups. Omitting `extracted/` caused the Source Viewer and Provenance citation modal to show \"Could not read source file\" after restore.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 10). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.9.3\n\n- **Backup now includes `hooks/` and `.synthadoc/extracted/`** — both directories were silently omitted from backups. `hooks/` contains user-customised hook scripts; `.synthadoc/extracted/` contains per-source text sidecars and PDF pagemaps written by the ingest agent. Without these, the Source Viewer and Provenance citation modal showed \"Could not read source file\" after restore.\n- **PyPI publish workflow** — `publish.yml` triggers on GitHub Release, builds the wheel with hatchling, and uploads to PyPI via OIDC trusted publishing (no API token stored). `pip install synthadoc` now works.\n- **Web UI bundled in pip package** — `synthadoc/data/web-ui/dist/` is now included in the wheel. `synthadoc web` previously returned 503 on pip-installed instances because the React dist was not packaged.\n- **CI sync checks** — PRs now fail if `synthadoc/data/obsidian-plugin/`, `synthadoc/data/web-ui/`, or `README-pypi.md` are out of sync with their source. All sync-check steps use `shell: bash` for Windows runner compatibility.\n- **`bump_version.py` extended** — now also updates `synthadoc/data/obsidian-plugin/manifest.json` alongside the source manifest, preventing version drift between the bundled copy and the plugin source.\n\n### v0.9.2\n\n- **`synthadoc/data/web-ui/dist/` packaged** — web UI dist files committed to the repo and bundled in the wheel via hatchling artifacts.\n- **`scripts/sync_web_ui.py`** — helper script to sync `web-ui/dist/` into `synthadoc/data/web-ui/dist/` after `npm run build`.\n\n### v0.9.1\n\n- **PyPI metadata** — added `readme`, `keywords`, `classifiers`, and `[project.urls]` to `pyproject.toml` for correct PyPI rendering.\n- **`README-pypi.md`** — generated from `README.md` by `scripts/prepare_pypi_readme.py`; strips `pypi-strip` blocks and rewrites relative links to absolute GitHub URLs.\n- **Branch protection compatibility** — removed post-merge `build-plugin` CI job that pushed compiled plugin directly to `main`. Plugin sync is now enforced in PRs: CI fails if `synthadoc/data/obsidian-plugin/` is out of sync.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n"}, "a6ee96f5d4144ee11f60725945227952af1279d6": {"t": 1785767280.140523, "n": 170679, "h": "75fe616320a31ff2", "c": "# Synthadoc — Design Document\n\n**Version:** 0.9.3 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent, checks CostGuard\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the summary (not raw text) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `skip` / `flag_contradiction`)\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads summary (not full text) + BM25 candidates + `purpose.md` scope. Outputs per-page action: `create`, `update`, `flag`, `skip` |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp}` |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 0.6.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports dangling slugs — pages listed in ROUTING.md that no longer exist in the wiki (enabled only when ROUTING.md exists). **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [--wiki-root ] — generate ROUTING.md from index.md branch structure\n│ ├── validate [--wiki-root ] — report dangling slugs and cross-branch duplicates\n│ └── clean [--wiki-root ] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [--wiki-root ]\n├── candidates\n│ ├── list [--wiki-root ]\n│ ├── promote |--all [--wiki-root ]\n│ └── discard |--all [--wiki-root ]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Log warning, continue _(configured but not yet enforced — cost_guard is wired to the config but check() is not called in the ingest path)_ |\n| `cost.hard_gate_usd` | float | `2.00` | Require explicit confirmation _(configured but not yet enforced — see above)_ |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × backup_count` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python scripts/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python scripts/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement (soft warn / hard gate) is implemented and configurable but not yet wired into the LLM call path — it is infrastructure ready to activate.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | Log warning; auto-continue |\n| `hard_gate_usd` | $2.00 | Prompt user `Proceed? [y/N]`; block if N; skip prompt if `auto_confirm=True` or `--yes` flag |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` at OS socket level. Not configurable. No remote access without a separate reverse proxy (which the user must explicitly set up).\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `--wiki-root `.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; a missing source file triggers `archived`. Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: if the source path recorded in frontmatter no longer exists on disk, the page is transitioned to `archived`.\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 10) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 10 # turns to include in each request (default: 10; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp}` or `{error}` | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `skill_registry.json`, `logs/`.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 10). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n"}, "d11681b944bb9c64d9d5302dde420d2b3473d302": {"t": 1785767283.3058074, "n": 170679, "h": "a2443b23dafd7497", "c": "# Synthadoc — Design Document\n\n**Version:** 0.9.2 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent, checks CostGuard\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the summary (not raw text) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `skip` / `flag_contradiction`)\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads summary (not full text) + BM25 candidates + `purpose.md` scope. Outputs per-page action: `create`, `update`, `flag`, `skip` |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp}` |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 0.6.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports dangling slugs — pages listed in ROUTING.md that no longer exist in the wiki (enabled only when ROUTING.md exists). **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [--wiki-root ] — generate ROUTING.md from index.md branch structure\n│ ├── validate [--wiki-root ] — report dangling slugs and cross-branch duplicates\n│ └── clean [--wiki-root ] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [--wiki-root ]\n├── candidates\n│ ├── list [--wiki-root ]\n│ ├── promote |--all [--wiki-root ]\n│ └── discard |--all [--wiki-root ]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── backup [-w wiki] [--output ] [--no-sources] [--no-exports] [--no-cache]\n├── restore [--name ] [--target ] [--port ]\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-WIKI-007` | Backup requires a newer `db_schema_version` than installed |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-QUERY-001` | LLM synthesis timed out; retry the query |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n| `ERR-PROV-001` | Daily API quota exhausted for today |\n| `ERR-PROV-002` | Coding tool CLI usage quota exhausted |\n| `ERR-AGENT-001` | LLM agent call failed (empty response, bad JSON, timeout) |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Log warning, continue _(configured but not yet enforced — cost_guard is wired to the config but check() is not called in the ingest path)_ |\n| `cost.hard_gate_usd` | float | `2.00` | Require explicit confirmation _(configured but not yet enforced — see above)_ |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × backup_count` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python scripts/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python scripts/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nFour independent cache layers:\n\n### Layer 1 — Vector embedding cache (`embeddings.db`) — optional\n\nStores fastembed vector embeddings for each wiki page, used to re-rank BM25 results by semantic similarity. Only active when `[search] vector = true` in `config.toml` and the `fastembed` optional dependency is installed (`pip install synthadoc[vectors]`). BM25 search is always computed in-memory and is never persisted to disk.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Query result cache (`cache.db` — `query_cache` table)\n\nStores full query answers keyed by `question + wiki_epoch + model`. The `wiki_epoch` is a monotonic counter incremented on every ingest or lifecycle state change, so any wiki update automatically invalidates all cached answers.\n\n**Cache key:**\n\n```python\ndef make_query_cache_key(question: str, epoch: int, model: str = \"\") -> str:\n normalized = \" \".join(question.lower().split()) # collapse whitespace, lowercase\n payload = f\"{normalized}|{epoch}|{model}\"\n return hashlib.sha256(payload.encode()).hexdigest()[:32]\n```\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Any `ingest` or lifecycle change | `wiki_epoch` incremented → all query cache entries for prior epoch bypassed |\n| `--no-cache` flag on query | Cache lookup skipped; fresh LLM call; result repopulated |\n| `cache clear` | Deletes all rows from both `response_cache` and `query_cache` tables |\n\n### Layer 4 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**Files:** `synthadoc/core/cost_guard.py`, `synthadoc/providers/pricing.py`\n\nCost tracking is live: `estimate_cost()` is called after each ingest and query operation and the result is written to `audit.db`. The `CostGuard` threshold enforcement (soft warn / hard gate) is implemented and configurable but not yet wired into the LLM call path — it is infrastructure ready to activate.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | Log warning; auto-continue |\n| `hard_gate_usd` | $2.00 | Prompt user `Proceed? [y/N]`; block if N; skip prompt if `auto_confirm=True` or `--yes` flag |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| Anthropic | claude-opus-4-7 | $0.000005 | $0.000025 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` at OS socket level. Not configurable. No remote access without a separate reverse proxy (which the user must explicitly set up).\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n assets/ ← primary resource dir; load with self.get_resource(\"format-notes.md\")\n format-notes.md\n references/ ← secondary resource dir (also searched by get_resource)\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n**Skill discovery priority:** `extra_dirs` (passed at runtime) → `/skills/` → `~/.synthadoc/skills/` → pip entry points (`synthadoc.skills` group) → built-ins. To distribute a skill as a pip package, declare an entry point pointing to the skill folder in your `pyproject.toml`.\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider` and wire it into `synthadoc/providers/__init__.py`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\nfrom typing import Optional\n\nclass MyProvider(LLMProvider):\n supports_vision: bool = False # set True only if the API accepts image inputs\n\n async def complete(\n self,\n messages: list[Message],\n system: Optional[str] = None,\n temperature: float = 0.0,\n max_tokens: int = 4096,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n\n async def complete_stream(self, messages, system=None, temperature=0.0, max_tokens=4096):\n \"\"\"Optional — override for streaming support.\"\"\"\n async for token in your_api_stream(...):\n yield token\n```\n\nAdd the provider name to `KNOWN_PROVIDERS` in `synthadoc/config.py` and add an `if name == \"my_provider\":` branch in `synthadoc/providers/__init__.py` that imports and returns an instance of your class.\n\n### Writing a hook\n\nHooks fire after key operations. They can be in any language; the process receives JSON on stdin and must exit 0 on success.\n\n**Available events:**\n\n| Event | Fired after | JSON context fields |\n|---|---|---|\n| `on_ingest_complete` | Every completed ingest job | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | Every completed lint run | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n**Register hooks in `.synthadoc/config.toml`:**\n\n```toml\n[hooks]\non_ingest_complete = \"hooks/notify.sh\" # non-blocking (default)\non_lint_complete = { cmd = \"hooks/alert.sh\", blocking = true } # blocking: error aborts the run\n```\n\n**Example hook script:**\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `--wiki-root `.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; a missing source file triggers `archived`. Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: if the source path recorded in frontmatter no longer exists on disk, the page is transitioned to `archived`.\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 10) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 10 # turns to include in each request (default: 10; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp}` or `{error}` | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── *.txt ← all batch ingest files at wiki root (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `skill_registry.json`, `logs/`.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": true,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"obsidian_plugin\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry, path exists | Hard stop — use `--name` to rename or `synthadoc uninstall` first |\n| Name in registry, path gone (stale) | Proceed with a printed note; stale entry is overwritten |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in `--output` (default: current directory). `restore` extracts the archive to `--target//` (default: same directory as the zip), rewrites host-specific config values (port, domain name), updates the global registry, re-applies scheduled jobs, and auto-reinstalls the Obsidian plugin if `obsidian_plugin` is `true` in the manifest. Both commands print a summary on completion.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 10). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n"}, "96c0fc73f00b8a616a32c4a2f7d8a6a41cb0a9c3": {"t": 1785767285.0206892, "n": 166931, "h": "2873b9cba36257e7", "c": "# Synthadoc — Design Document\n\n**Version:** 0.9.2 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent, checks CostGuard\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the summary (not raw text) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `skip` / `flag_contradiction`)\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads summary (not full text) + BM25 candidates + `purpose.md` scope. Outputs per-page action: `create`, `update`, `flag`, `skip` |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp}` |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 0.6.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports dangling slugs — pages listed in ROUTING.md that no longer exist in the wiki (enabled only when ROUTING.md exists). **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [--wiki-root ] — generate ROUTING.md from index.md branch structure\n│ ├── validate [--wiki-root ] — report dangling slugs and cross-branch duplicates\n│ └── clean [--wiki-root ] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [--wiki-root ]\n├── candidates\n│ ├── list [--wiki-root ]\n│ ├── promote |--all [--wiki-root ]\n│ └── discard |--all [--wiki-root ]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Log warning, continue _(configured but not yet enforced — cost_guard is wired to the config but check() is not called in the ingest path)_ |\n| `cost.hard_gate_usd` | float | `2.00` | Require explicit confirmation _(configured but not yet enforced — see above)_ |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × backup_count` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python scripts/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python scripts/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nThree independent cache layers:\n\n### Layer 1 — Embedding cache (`embeddings.db`)\n\nStores the BM25 index entry for each wiki page, keyed by page content SHA-256. When a page is updated, only that page's entry is refreshed.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**File:** `synthadoc/core/cost_guard.py`\n\nEnforces per-operation budget limits. Evaluated before every LLM call.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | Log warning; auto-continue |\n| `hard_gate_usd` | $2.00 | Prompt user `Proceed? [y/N]`; block if N; skip prompt if `auto_confirm=True` or `--yes` flag |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` at OS socket level. Not configurable. No remote access without a separate reverse proxy (which the user must explicitly set up).\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n references/\n format-notes.md ← optional; load with self.get_resource(\"format-notes.md\")\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\n\nclass MyProvider(LLMProvider):\n\n async def complete(\n self,\n messages: list[Message],\n model: str,\n temperature: float = 0.0,\n **kwargs,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n```\n\nPlace in `~/.synthadoc/providers/` or the wiki `providers/` directory. Reference by name in config:\n\n```toml\n[agents]\ndefault = { provider = \"my_provider\", model = \"my-model-id\" }\n```\n\n### Writing a hook\n\nHooks can be in any language. They receive JSON on stdin and must exit 0 on success.\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `--wiki-root `.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; a missing source file triggers `archived`. Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: if the source path recorded in frontmatter no longer exists on disk, the page is transitioned to `archived`.\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 10) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 10 # turns to include in each request (default: 10; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp}` or `{error}` | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── sources.txt ← batch ingest manifest (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `skill_registry.json`, `logs/`.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": false,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry | Hard stop — use `--name` or uninstall first |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in the output directory (default: current directory). `restore` extracts the archive, rewrites host-specific config values (port, domain name), updates the global registry, and re-applies any scheduled jobs. Both commands print a summary on completion; `restore` also prints a post-restore checklist.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 10). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n"}, "eb6bbcbcaec5ca0e048ce384945453dc5b1b7a4f": {"t": 1785767286.9727142, "n": 166931, "h": "2fbe51156a1a38b7", "c": "# Synthadoc — Design Document\n\n**Version:** 0.9.1 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent, checks CostGuard\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the summary (not raw text) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `skip` / `flag_contradiction`)\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads summary (not full text) + BM25 candidates + `purpose.md` scope. Outputs per-page action: `create`, `update`, `flag`, `skip` |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp}` |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 0.6.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports dangling slugs — pages listed in ROUTING.md that no longer exist in the wiki (enabled only when ROUTING.md exists). **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [--wiki-root ] — generate ROUTING.md from index.md branch structure\n│ ├── validate [--wiki-root ] — report dangling slugs and cross-branch duplicates\n│ └── clean [--wiki-root ] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [--wiki-root ]\n├── candidates\n│ ├── list [--wiki-root ]\n│ ├── promote |--all [--wiki-root ]\n│ └── discard |--all [--wiki-root ]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Log warning, continue _(configured but not yet enforced — cost_guard is wired to the config but check() is not called in the ingest path)_ |\n| `cost.hard_gate_usd` | float | `2.00` | Require explicit confirmation _(configured but not yet enforced — see above)_ |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × backup_count` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python scripts/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python scripts/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nThree independent cache layers:\n\n### Layer 1 — Embedding cache (`embeddings.db`)\n\nStores the BM25 index entry for each wiki page, keyed by page content SHA-256. When a page is updated, only that page's entry is refreshed.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**File:** `synthadoc/core/cost_guard.py`\n\nEnforces per-operation budget limits. Evaluated before every LLM call.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | Log warning; auto-continue |\n| `hard_gate_usd` | $2.00 | Prompt user `Proceed? [y/N]`; block if N; skip prompt if `auto_confirm=True` or `--yes` flag |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` at OS socket level. Not configurable. No remote access without a separate reverse proxy (which the user must explicitly set up).\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n references/\n format-notes.md ← optional; load with self.get_resource(\"format-notes.md\")\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\n\nclass MyProvider(LLMProvider):\n\n async def complete(\n self,\n messages: list[Message],\n model: str,\n temperature: float = 0.0,\n **kwargs,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n```\n\nPlace in `~/.synthadoc/providers/` or the wiki `providers/` directory. Reference by name in config:\n\n```toml\n[agents]\ndefault = { provider = \"my_provider\", model = \"my-model-id\" }\n```\n\n### Writing a hook\n\nHooks can be in any language. They receive JSON on stdin and must exit 0 on success.\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `--wiki-root `.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; a missing source file triggers `archived`. Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: if the source path recorded in frontmatter no longer exists on disk, the page is transitioned to `archived`.\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 10) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 10 # turns to include in each request (default: 10; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp}` or `{error}` | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── sources.txt ← batch ingest manifest (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `skill_registry.json`, `logs/`.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": false,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry | Hard stop — use `--name` or uninstall first |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in the output directory (default: current directory). `restore` extracts the archive, rewrites host-specific config values (port, domain name), updates the global registry, and re-applies any scheduled jobs. Both commands print a summary on completion; `restore` also prints a post-restore checklist.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 10). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n"}, "ad21c65010f6a00e4d8467eb0b8da413118968dd": {"t": 1785767289.0844035, "n": 166931, "h": "8e71c8db6ce5d384", "c": "# Synthadoc — Design Document\n\n**Version:** 0.9.0 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n28. [Backup & Restore](#28-backup--restore)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent, checks CostGuard\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the summary (not raw text) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `skip` / `flag_contradiction`)\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads summary (not full text) + BM25 candidates + `purpose.md` scope. Outputs per-page action: `create`, `update`, `flag`, `skip` |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp}` |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 0.6.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports dangling slugs — pages listed in ROUTING.md that no longer exist in the wiki (enabled only when ROUTING.md exists). **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [--wiki-root ] — generate ROUTING.md from index.md branch structure\n│ ├── validate [--wiki-root ] — report dangling slugs and cross-branch duplicates\n│ └── clean [--wiki-root ] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [--wiki-root ]\n├── candidates\n│ ├── list [--wiki-root ]\n│ ├── promote |--all [--wiki-root ]\n│ └── discard |--all [--wiki-root ]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Log warning, continue _(configured but not yet enforced — cost_guard is wired to the config but check() is not called in the ingest path)_ |\n| `cost.hard_gate_usd` | float | `2.00` | Require explicit confirmation _(configured but not yet enforced — see above)_ |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × backup_count` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python scripts/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python scripts/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nThree independent cache layers:\n\n### Layer 1 — Embedding cache (`embeddings.db`)\n\nStores the BM25 index entry for each wiki page, keyed by page content SHA-256. When a page is updated, only that page's entry is refreshed.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**File:** `synthadoc/core/cost_guard.py`\n\nEnforces per-operation budget limits. Evaluated before every LLM call.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | Log warning; auto-continue |\n| `hard_gate_usd` | $2.00 | Prompt user `Proceed? [y/N]`; block if N; skip prompt if `auto_confirm=True` or `--yes` flag |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` at OS socket level. Not configurable. No remote access without a separate reverse proxy (which the user must explicitly set up).\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n references/\n format-notes.md ← optional; load with self.get_resource(\"format-notes.md\")\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\n\nclass MyProvider(LLMProvider):\n\n async def complete(\n self,\n messages: list[Message],\n model: str,\n temperature: float = 0.0,\n **kwargs,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n```\n\nPlace in `~/.synthadoc/providers/` or the wiki `providers/` directory. Reference by name in config:\n\n```toml\n[agents]\ndefault = { provider = \"my_provider\", model = \"my-model-id\" }\n```\n\n### Writing a hook\n\nHooks can be in any language. They receive JSON on stdin and must exit 0 on success.\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `--wiki-root `.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; a missing source file triggers `archived`. Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: if the source path recorded in frontmatter no longer exists on disk, the page is transitioned to `archived`.\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 10) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 10 # turns to include in each request (default: 10; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp}` or `{error}` | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## 28. Backup & Restore\n\nThe `synthadoc backup` and `synthadoc restore` commands package a running wiki domain into a portable compressed zip and re-register it on any machine.\n\n### Architecture\n\nAll file I/O is handled by a dedicated backup engine (pure stdlib — no new pip dependencies). The CLI layer manages Typer commands, registry operations, and interactive prompts, reusing existing port-allocation and registry helpers from the installation subsystem.\n\n### Zip structure\n\n```\nsynthadoc-backup--.zip\n├── manifest.json ← always present; last entry wins if duplicated\n├── AGENTS.md ← LLM agent instructions (if present)\n├── ROUTING.md ← query routing index (if present)\n├── log.md ← human-readable activity log (if present)\n├── sources.txt ← batch ingest manifest (if present)\n├── wiki/\n│ ├── *.md\n│ └── candidates/*.md\n├── .synthadoc/\n│ ├── config.toml\n│ ├── audit.db\n│ └── cache.db ← included by default; skip with --no-cache\n├── exports/ ← included by default; skip with --no-exports\n└── raw_sources/ ← included by default; skip with --no-sources\n```\n\nAlways excluded: `jobs.db`, `embeddings.db`, `server.pid`, `skill_registry.json`, `logs/`.\n\n### Manifest\n\nEvery backup contains a `manifest.json` at the zip root:\n\n```json\n{\n \"synthadoc_version\": \"1.0.0\",\n \"db_schema_version\": 1,\n \"cache_version\": \"4\",\n \"wiki_name\": \"history-of-computing\",\n \"backed_up_at\": \"2026-06-24T10:30:00Z\",\n \"source_os\": \"windows\",\n \"source_hostname\": \"dev-machine\",\n \"page_count\": 87,\n \"includes_sources\": false,\n \"includes_exports\": true,\n \"includes_cache\": true,\n \"checksum_sha256\": \"abc123...\"\n}\n```\n\n`db_schema_version` is read from SQLite `PRAGMA user_version` in `audit.db`. The restore tool aborts if the backup's version exceeds the installed version. `checksum_sha256` is the SHA-256 of all non-manifest zip members in sorted name order.\n\n### Restore conflict rules\n\n| Situation | Behaviour |\n|---|---|\n| Name not in registry | Register normally |\n| Name in registry | Hard stop — use `--name` or uninstall first |\n| Demo wiki renamed via `--name` | Warn + `y/N` prompt (breaks `demo sync`) |\n| Port taken | System suggests the next available port; user confirms or overrides |\n\n### CLI commands\n\n```\nsynthadoc backup -w [--output ] [--no-sources] [--no-exports] [--no-cache]\n\nsynthadoc restore [--name ] [--target ] [--port ]\n```\n\n`backup` creates a timestamped `ZIP_DEFLATED` archive in the output directory (default: current directory). `restore` extracts the archive, rewrites host-specific config values (port, domain name), updates the global registry, and re-applies any scheduled jobs. Both commands print a summary on completion; `restore` also prints a post-restore checklist.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 10). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n"}, "de6d34cdbc0b6ac2c484cf57be4d19d9595d4eae": {"t": 1785767290.909197, "n": 164012, "h": "05217cbddbae0cce", "c": "# Synthadoc — Design Document\n\n**Version:** 0.9.0 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent, checks CostGuard\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the summary (not raw text) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `skip` / `flag_contradiction`)\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads summary (not full text) + BM25 candidates + `purpose.md` scope. Outputs per-page action: `create`, `update`, `flag`, `skip` |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp}` |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 0.6.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports dangling slugs — pages listed in ROUTING.md that no longer exist in the wiki (enabled only when ROUTING.md exists). **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [--wiki-root ] — generate ROUTING.md from index.md branch structure\n│ ├── validate [--wiki-root ] — report dangling slugs and cross-branch duplicates\n│ └── clean [--wiki-root ] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [--wiki-root ]\n├── candidates\n│ ├── list [--wiki-root ]\n│ ├── promote |--all [--wiki-root ]\n│ └── discard |--all [--wiki-root ]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Log warning, continue _(configured but not yet enforced — cost_guard is wired to the config but check() is not called in the ingest path)_ |\n| `cost.hard_gate_usd` | float | `2.00` | Require explicit confirmation _(configured but not yet enforced — see above)_ |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × backup_count` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python scripts/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python scripts/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nThree independent cache layers:\n\n### Layer 1 — Embedding cache (`embeddings.db`)\n\nStores the BM25 index entry for each wiki page, keyed by page content SHA-256. When a page is updated, only that page's entry is refreshed.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**File:** `synthadoc/core/cost_guard.py`\n\nEnforces per-operation budget limits. Evaluated before every LLM call.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | Log warning; auto-continue |\n| `hard_gate_usd` | $2.00 | Prompt user `Proceed? [y/N]`; block if N; skip prompt if `auto_confirm=True` or `--yes` flag |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` at OS socket level. Not configurable. No remote access without a separate reverse proxy (which the user must explicitly set up).\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n references/\n format-notes.md ← optional; load with self.get_resource(\"format-notes.md\")\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\n\nclass MyProvider(LLMProvider):\n\n async def complete(\n self,\n messages: list[Message],\n model: str,\n temperature: float = 0.0,\n **kwargs,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n```\n\nPlace in `~/.synthadoc/providers/` or the wiki `providers/` directory. Reference by name in config:\n\n```toml\n[agents]\ndefault = { provider = \"my_provider\", model = \"my-model-id\" }\n```\n\n### Writing a hook\n\nHooks can be in any language. They receive JSON on stdin and must exit 0 on success.\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `--wiki-root `.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; a missing source file triggers `archived`. Manual transitions follow a defined graph (see Transition rules below); only semantically valid paths are permitted. Lint and ingest write state directly and are not subject to the graph restriction.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\nAutomated transitions (lint, ingest) write state directly and are not subject to the graph. User-driven transitions (CLI, HTTP API, MCP) are validated against `ALLOWED_LIFECYCLE_TRANSITIONS` in `synthadoc/storage/wiki.py`; invalid paths are rejected with HTTP 422 or an MCP error dict.\n\n| From | To | Trigger | Notes |\n|---|---|---|---|\n| _(none)_ | `draft` | ingest | New page created by ingest |\n| `draft` | `active` | lint, cli/api/mcp | Lint auto-promotes when all checks pass; or manual activate |\n| `draft` | `archived` | cli/api/mcp | Abandon a draft without publishing |\n| `active` | `contradicted` | lint, cli/api/mcp | Lint detects conflict automatically; or user manually flags |\n| `active` | `stale` | lint | Local source: SHA-256 hash mismatch; URL source older than `url_staleness_days` |\n| `active` | `archived` | lint, cli/api/mcp | Lint: local source missing or URL 404/410; or manual retire |\n| `stale` | `draft` | cli/api/mcp | Revise stale content — puts page back in review queue |\n| `stale` | `active` | cli/api/mcp | Re-validate without revision — user confirms content still accurate |\n| `stale` | `archived` | lint, cli/api/mcp | Lint: source gone; or manual archive |\n| `contradicted` | `draft` | cli/api/mcp | Revise contradicted content — resets to review queue |\n| `contradicted` | `active` | cli/api/mcp | Resolve contradiction and re-activate directly |\n| `contradicted` | `archived` | cli/api/mcp | Archive after reviewing the conflict |\n| `archived` | `draft` | cli/api/mcp | Restore for revision — places page back in review queue |\n\nTransitions not in this table are rejected. Notable blocked paths: `stale ↔ contradicted` (different issue types that should not be crossed directly), `archived → active/stale/contradicted` (must go through `draft` for re-review first), `draft → stale/contradicted` (unpublished pages cannot be in those states).\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: if the source path recorded in frontmatter no longer exists on disk, the page is transitioned to `archived`.\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 10) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 10 # turns to include in each request (default: 10; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp}` or `{error}` | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 10). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n"}, "d1b14bbe290ceb4ff8a9ba2b01417853b8223092": {"t": 1785767292.7210488, "n": 162924, "h": "f1fd171cd02930cd", "c": "# Synthadoc — Design Document\n\n**Version:** 0.9.0 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent, checks CostGuard\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the summary (not raw text) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `skip` / `flag_contradiction`)\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads summary (not full text) + BM25 candidates + `purpose.md` scope. Outputs per-page action: `create`, `update`, `flag`, `skip` |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp}` |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 0.6.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports dangling slugs — pages listed in ROUTING.md that no longer exist in the wiki (enabled only when ROUTING.md exists). **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [--wiki-root ] — generate ROUTING.md from index.md branch structure\n│ ├── validate [--wiki-root ] — report dangling slugs and cross-branch duplicates\n│ └── clean [--wiki-root ] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [--wiki-root ]\n├── candidates\n│ ├── list [--wiki-root ]\n│ ├── promote |--all [--wiki-root ]\n│ └── discard |--all [--wiki-root ]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Log warning, continue _(configured but not yet enforced — cost_guard is wired to the config but check() is not called in the ingest path)_ |\n| `cost.hard_gate_usd` | float | `2.00` | Require explicit confirmation _(configured but not yet enforced — see above)_ |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × backup_count` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python scripts/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python scripts/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nThree independent cache layers:\n\n### Layer 1 — Embedding cache (`embeddings.db`)\n\nStores the BM25 index entry for each wiki page, keyed by page content SHA-256. When a page is updated, only that page's entry is refreshed.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**File:** `synthadoc/core/cost_guard.py`\n\nEnforces per-operation budget limits. Evaluated before every LLM call.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | Log warning; auto-continue |\n| `hard_gate_usd` | $2.00 | Prompt user `Proceed? [y/N]`; block if N; skip prompt if `auto_confirm=True` or `--yes` flag |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` at OS socket level. Not configurable. No remote access without a separate reverse proxy (which the user must explicitly set up).\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n references/\n format-notes.md ← optional; load with self.get_resource(\"format-notes.md\")\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\n\nclass MyProvider(LLMProvider):\n\n async def complete(\n self,\n messages: list[Message],\n model: str,\n temperature: float = 0.0,\n **kwargs,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n```\n\nPlace in `~/.synthadoc/providers/` or the wiki `providers/` directory. Reference by name in config:\n\n```toml\n[agents]\ndefault = { provider = \"my_provider\", model = \"my-model-id\" }\n```\n\n### Writing a hook\n\nHooks can be in any language. They receive JSON on stdin and must exit 0 on success.\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `--wiki-root `.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; a missing source file triggers `archived`. Manual transitions allow operators to override any state.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\n| From | To | Trigger | Condition |\n|---|---|---|---|\n| _(none)_ | `draft` | `ingest` | New page created by ingest |\n| `draft` | `active` | `lint` | Page passes all lint checks |\n| `draft` / `stale` / `contradicted` | `active` | `cli` / `api` | Manual activate (`synthadoc lifecycle activate`) |\n| `active` | `contradicted` | `lint` | Contradiction detected between two or more sources |\n| `active` | `stale` | `lint` | Local source: SHA-256 hash mismatch; or URL source older than `url_staleness_days` |\n| `active` / `stale` | `archived` | `lint` | Local source no longer exists on disk; or URL source returns 404/410 (opt-in) |\n| any | `archived` | `cli` / `api` | Manual archive (`synthadoc lifecycle archive`) |\n| `archived` | `draft` | `cli` / `api` | Manual restore (`synthadoc lifecycle restore`) — places page back in review queue |\n| `contradicted` | `archived` | `cli` / `api` | Manual archive after reviewing the conflict |\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: if the source path recorded in frontmatter no longer exists on disk, the page is transitioned to `archived`.\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 10) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 10 # turns to include in each request (default: 10; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags, lint_warnings, sources}` or `{error, slug}` | Claude only |\n| `synthadoc_list_pages` | `status?: str` (default `\"all\"`) | `{pages: [{slug, title, status, type, has_sources}], total: int}` | Neither |\n| `synthadoc_context` | `goal: str`, `token_budget?: int` (default `10000`) | `{goal, token_budget, tokens_used, pages: [{slug, relevance, excerpt, source, confidence, tags, estimated_tokens}], omitted: [{slug, estimated_tokens}]}` | Neither |\n| `synthadoc_export` | `format?: str` (default `\"okf\"`), `output_path?: str` (okf defaults to `/exports/-okf-/`), `status_filter?: str` (default `\"all\"`) | okf writes folder to disk → `{format, output_path, files_written, pages}`. Other formats: with `output_path` → `{format, output_path, pages}`; without → `{format, content, pages}`. Formats: `okf`, `llms.txt`, `llms-full.txt`, `json`, `graphml` | Neither |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp}` or `{error}` | Neither |\n| `synthadoc_lint_report` | *(none)* | `{contradicted: [str], orphans: [str], adversarial_warnings: int, adversarial_pages: [str]}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{job_id, scope}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs` and `synthadoc_list_pages`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n`synthadoc_lint` enqueues a background LLM analysis job; use `synthadoc_jobs` to poll progress. `synthadoc_lint_report` is the zero-cost alternative — reads current contradiction/orphan state from wiki files instantly, no job enqueued.\n\n`synthadoc_read_page` returns `sources` as `[{file, ingested}]` and `lint_warnings` as a list of adversarial warning strings — empty list when clean.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 10). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 12 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_list_pages`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_lint_report`, `synthadoc_context`, `synthadoc_export`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n- **OKF `type:` field** — IngestAgent now writes a `type:` frontmatter field on every compiled page (values: `concept`, `person`, `organization`, `technology`, `event`, `location`, `product`). The field is required by OKF v0.1 and enables type-grouped `index.md` in the export bundle. Pages ingested before v0.9.0 can be backfilled via `synthadoc demo sync` (demo wikis) or re-running `synthadoc ingest` (custom wikis).\n- **`synthadoc demo sync` — optional wiki name** — running `synthadoc demo sync` without a wiki name argument syncs all registered demo wikis in one pass. The sync step also backfills `type:` on existing pages that were compiled before v0.9.0 without the field.\n- **SSE shutdown stability** — a log filter installed on `uvicorn.error` at startup suppresses three benign error classes that appear when the server exits while SSE connections are open: `asyncio.CancelledError`, `KeyboardInterrupt`, and the `RuntimeError(\"Expected ASGI message 'http.response.body'…\")` that Starlette's error middleware raises after cancellation. Actual errors during normal operation are unaffected.\n"}, "da5d3e67592d865c978f6883f4b7d190c4dfadd4": {"t": 1785767294.642247, "n": 160362, "h": "afba6262e41758d8", "c": "# Synthadoc — Design Document\n\n**Version:** 0.8.0 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n27. [MCP Server](#27-mcp-server)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared. The MCP layer positions Synthadoc as persistent domain memory: an AI client (Claude Desktop, Claude Code) acts as the reasoning brain while Synthadoc handles BM25 search, lifecycle, and the immutable audit trail.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent, checks CostGuard\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the summary (not raw text) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `skip` / `flag_contradiction`)\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads summary (not full text) + BM25 candidates + `purpose.md` scope. Outputs per-page action: `create`, `update`, `flag`, `skip` |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp}` |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 0.6.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports dangling slugs — pages listed in ROUTING.md that no longer exist in the wiki (enabled only when ROUTING.md exists). **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [--wiki-root ] — generate ROUTING.md from index.md branch structure\n│ ├── validate [--wiki-root ] — report dangling slugs and cross-branch duplicates\n│ └── clean [--wiki-root ] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [--wiki-root ]\n├── candidates\n│ ├── list [--wiki-root ]\n│ ├── promote |--all [--wiki-root ]\n│ └── discard |--all [--wiki-root ]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Log warning, continue _(configured but not yet enforced — cost_guard is wired to the config but check() is not called in the ingest path)_ |\n| `cost.hard_gate_usd` | float | `2.00` | Require explicit confirmation _(configured but not yet enforced — see above)_ |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × backup_count` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python scripts/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python scripts/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nThree independent cache layers:\n\n### Layer 1 — Embedding cache (`embeddings.db`)\n\nStores the BM25 index entry for each wiki page, keyed by page content SHA-256. When a page is updated, only that page's entry is refreshed.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**File:** `synthadoc/core/cost_guard.py`\n\nEnforces per-operation budget limits. Evaluated before every LLM call.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | Log warning; auto-continue |\n| `hard_gate_usd` | $2.00 | Prompt user `Proceed? [y/N]`; block if N; skip prompt if `auto_confirm=True` or `--yes` flag |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` at OS socket level. Not configurable. No remote access without a separate reverse proxy (which the user must explicitly set up).\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n references/\n format-notes.md ← optional; load with self.get_resource(\"format-notes.md\")\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\n\nclass MyProvider(LLMProvider):\n\n async def complete(\n self,\n messages: list[Message],\n model: str,\n temperature: float = 0.0,\n **kwargs,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n```\n\nPlace in `~/.synthadoc/providers/` or the wiki `providers/` directory. Reference by name in config:\n\n```toml\n[agents]\ndefault = { provider = \"my_provider\", model = \"my-model-id\" }\n```\n\n### Writing a hook\n\nHooks can be in any language. They receive JSON on stdin and must exit 0 on success.\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `--wiki-root `.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; a missing source file triggers `archived`. Manual transitions allow operators to override any state.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\n| From | To | Trigger | Condition |\n|---|---|---|---|\n| _(none)_ | `draft` | `ingest` | New page created by ingest |\n| `draft` | `active` | `lint` | Page passes all lint checks |\n| `draft` / `stale` / `contradicted` | `active` | `cli` / `api` | Manual activate (`synthadoc lifecycle activate`) |\n| `active` | `contradicted` | `lint` | Contradiction detected between two or more sources |\n| `active` | `stale` | `lint` | Local source: SHA-256 hash mismatch; or URL source older than `url_staleness_days` |\n| `active` / `stale` | `archived` | `lint` | Local source no longer exists on disk; or URL source returns 404/410 (opt-in) |\n| any | `archived` | `cli` / `api` | Manual archive (`synthadoc lifecycle archive`) |\n| `archived` | `draft` | `cli` / `api` | Manual restore (`synthadoc lifecycle restore`) — places page back in review queue |\n| `contradicted` | `archived` | `cli` / `api` | Manual archive after reviewing the conflict |\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: if the source path recorded in frontmatter no longer exists on disk, the page is transitioned to `archived`.\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 10) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 10 # turns to include in each request (default: 10; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## 27. MCP Server\n\nSynthadoc exposes its core operations as an MCP (Model Context Protocol) server, allowing AI agents — Claude Desktop, Claude Code, n8n, LangGraph, or any MCP-compliant host — to read, write, and manage the wiki without running Synthadoc's own query LLM.\n\n### Architecture\n\nThe MCP server is implemented with FastMCP and mounted at `/mcp/sse` on the existing HTTP server via ASGI mount. No extra port or process is required — the MCP endpoint and the HTTP REST API share the same Orchestrator singleton and storage layer.\n\n```\nsynthadoc serve -w my-wiki --port 7070\n\n ┌────────────────────────────────────────────┐\n │ HTTP server (Starlette) :7070 │\n │ GET /query/stream → QueryAgent │\n │ POST /ingest → IngestAgent │\n │ GET /app → Web Chat UI │\n │ /mcp (ASGI mount) → FastMCP │\n │ /mcp/sse → SSE transport │\n └────────────────────────────────────────────┘\n ↑ shared Orchestrator + WikiStorage\n```\n\n### Transport options\n\n| Client | Transport | Config mechanism |\n|---|---|---|\n| Claude Desktop | stdio | `command` + `args` in `mcpServers` JSON |\n| Claude Code CLI | SSE (`--transport sse`) | `claude mcp add --transport sse ` |\n| n8n, LangGraph, custom agents | HTTP/SSE | Direct HTTP connection to `/mcp/sse` |\n\nClaude Desktop does not support `\"url\"`-based HTTP connections in its `mcpServers` config — stdio is the only supported transport. Claude Code supports both SSE and stdio. The SSE endpoint path is exactly `/mcp/sse` (not `/mcp` or `/mcp/`).\n\nThe MCP and HTTP servers both bind to `127.0.0.1` — remote access requires an explicit reverse proxy (user-managed).\n\n### Multi-wiki naming\n\nWhen a wiki root is provided, the FastMCP server name is set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing` for a wiki rooted at `history-of-computing/`). This name appears in Claude Desktop's connected-servers UI and in `claude mcp list` output.\n\nEvery tool description is automatically prefixed with `Wiki: {wiki-name}. ` at server startup. This allows Claude to route tool calls correctly when multiple Synthadoc servers are connected simultaneously — Claude reads the prefix in the tool description and routes to the matching wiki.\n\nFor Claude Desktop, `mcpServers` key names must use underscores (e.g. `synthadoc_history_of_computing`) — hyphens cause a load failure.\n\n### Tool reference\n\n| Tool | Parameters | Returns | LLM cost |\n|---|---|---|---|\n| `synthadoc_search` | `terms: str` | `{results: [{slug, score, title, snippet}]}` | Claude only |\n| `synthadoc_read_page` | `slug: str` | `{slug, title, content, status, type, tags}` or `{error, slug}` | Claude only |\n| `synthadoc_write_page` | `slug: str`, `content: str`, `title?: str` | `{slug, title, status}` or `{error, slug}` | Neither |\n| `synthadoc_status` | *(none)* | `{pages: int, wiki: str}` | Neither |\n| `synthadoc_jobs` | `status?: str` (default `\"all\"`) | `{jobs: [{id, operation, status, created, source?, error?}]}` | Neither |\n| `synthadoc_lifecycle` | `slug: str`, `to_state: str`, `reason: str` | `{slug, from_state, to_state, reason, timestamp}` or `{error}` | Neither |\n| `synthadoc_ingest` | `source: str` | `{job_id, source}` | Synthadoc |\n| `synthadoc_lint` | `scope?: str` (default `\"all\"`) | `{contradictions_found: int, orphans: [str]}` | Synthadoc |\n\nValid `to_state` values for `synthadoc_lifecycle`: `active`, `draft`, `stale`, `contradicted`, `archived`.\n\nValid `status` values for `synthadoc_jobs`: `all`, `pending`, `running`, `completed`, `failed`, `skipped`, `cancelled`, `dead`. `running` maps to the internal `in_progress` state.\n\n`synthadoc_write_page` clears `contradiction_note` and bumps the wiki epoch (invalidating the query cache). It does not change `status` — use `synthadoc_lifecycle` to transition state after editing.\n\n### Brain/memory architecture\n\nThe MCP integration separates reasoning from persistence:\n\n| Layer | Role | What it handles |\n|---|---|---|\n| Claude (Desktop or Code) | **Brain** — reasoning, synthesis, editorial judgment | Tool chaining, cross-domain inference, writing quality |\n| Synthadoc MCP | **Memory** — domain knowledge, lifecycle, audit | BM25 search, page storage, 5-state lifecycle, immutable event log |\n\nPractical consequences:\n\n- **No double-LLM cost** — `synthadoc_search` and `synthadoc_read_page` return raw data; Claude does the synthesis. Only `synthadoc_ingest` and `synthadoc_lint` call Synthadoc's configured LLM.\n- **Claude handles editorial quality** — synthesising contradictions, deciding what is authoritative, drafting resolved content.\n- **Synthadoc handles auditability** — every write goes through `WikiStorage.write_page()`; every lifecycle transition is recorded in `audit.db` with `triggered_by = mcp`, a timestamp, and the stated reason.\n- **Dynamic vs. static hints** — Claude's next tool call is driven by its own reasoning (dynamic); Synthadoc's `HintEngine` in the web UI is static (predefined patterns). This gap is a feature: Claude handles the editorial reasoning that would require complex heuristics to encode statically.\n\n### Contradiction resolution via MCP\n\nThe canonical use case is MCP-driven contradiction resolution (documented in the user quick-start guide, Step 9, Option 3):\n\n```\nsynthadoc_read_page(\"grace-hopper\")\n → returns page with status: contradicted, contradiction_note: \"...\"\n\nsynthadoc_write_page(slug=\"grace-hopper\", content=\"\")\n → clears contradiction_note, bumps epoch\n\nsynthadoc_lifecycle(slug=\"grace-hopper\", to_state=\"active\",\n reason=\"Resolved: both views preserved, A-0 attribution corrected\")\n → audit.db: slug=grace-hopper, from=contradicted, to=active, triggered_by=mcp\n```\n\nThe audit trail records the same fields as a manual CLI transition — the MCP path is a first-class lifecycle actor.\n\n### CLI flags\n\n| Flag | Effect |\n|---|---|\n| `--mcp-only` | Start only the MCP endpoint; suppress HTTP REST API and web UI |\n| `--http-only` | Start only the HTTP server; suppress the MCP mount |\n\nDefault (no flag): both MCP and HTTP start together on the same port.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 10). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n\n### v0.9.0 (Community Edition)\n\n- **MCP server — 8 tools** — Synthadoc exposes `synthadoc_search`, `synthadoc_read_page`, `synthadoc_write_page`, `synthadoc_status`, `synthadoc_jobs`, `synthadoc_lifecycle`, `synthadoc_ingest`, and `synthadoc_lint` via the Model Context Protocol. Mounted at `/mcp/sse` on the existing HTTP server (no extra port or process). Tools share the same Orchestrator singleton as the HTTP REST API.\n- **`synthadoc_write_page`** — lifecycle-aware content editing: updates page body, clears `contradiction_note`, bumps the wiki epoch (cache invalidation). Proper MCP alternative to writing wiki files directly — every edit goes through `WikiStorage.write_page()` and is query-cache-coherent.\n- **Multi-wiki server naming** — FastMCP server name auto-set to `synthadoc-{wiki-name}` (e.g. `synthadoc-history-of-computing`). All tool descriptions prefixed with `Wiki: {wiki-name}.` at startup so Claude can route correctly when multiple Synthadoc servers are connected simultaneously.\n- **Transport support** — stdio (Claude Desktop), SSE via `--transport sse` (Claude Code CLI), HTTP/SSE direct connection (n8n, LangGraph, custom agents). Claude Desktop requires underscores in `mcpServers` key names (hyphens cause load failure).\n- **Brain/memory architecture** — Claude acts as the reasoning brain (editorial judgment, synthesis, tool chaining); Synthadoc MCP acts as persistent domain memory (BM25 search, 5-state lifecycle, immutable audit trail). `synthadoc_search` and `synthadoc_read_page` return raw data with no Synthadoc LLM call; only `synthadoc_ingest` and `synthadoc_lint` consume tokens from the configured provider.\n- **`--mcp-only` / `--http-only` serve flags** — deploy MCP-only (no web UI or REST API) or HTTP-only (no MCP mount) for constrained environments.\n"}, "fc361dc7df45baf05a36c21bb15b42ed7e44877c": {"t": 1785767296.969478, "n": 152157, "h": "b3c602c1996cdda2", "c": "# Synthadoc — Design Document\n\n**Version:** 0.8.0 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent, checks CostGuard\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the summary (not raw text) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `skip` / `flag_contradiction`)\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads summary (not full text) + BM25 candidates + `purpose.md` scope. Outputs per-page action: `create`, `update`, `flag`, `skip` |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of five formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n| `okf` | [OKF v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory — one Markdown file per page with conformant YAML frontmatter (`type`, `title`, `description`, `resource`, `tags`, `timestamp`); `index.md` grouped by knowledge type; `log.md` lifecycle history; `[[wikilinks]]` rewritten to relative OKF paths |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset. For `okf`, the accepted values are `all` (active + contradicted, the default) or `active` only — draft, stale, and archived pages are always excluded from OKF bundles regardless of the flag.\n\n**OKF return type:** Unlike other formats (which return a single string), `okf` returns `dict[str, str]` — a map of relative file paths to file contents. The HTTP endpoint serialises this as a JSON manifest; the CLI writes the manifest as a directory tree. `--output` is required for `okf`.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp}` |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 0.6.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports dangling slugs — pages listed in ROUTING.md that no longer exist in the wiki (enabled only when ROUTING.md exists). **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [--wiki-root ] — generate ROUTING.md from index.md branch structure\n│ ├── validate [--wiki-root ] — report dangling slugs and cross-branch duplicates\n│ └── clean [--wiki-root ] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [--wiki-root ]\n├── candidates\n│ ├── list [--wiki-root ]\n│ ├── promote |--all [--wiki-root ]\n│ └── discard |--all [--wiki-root ]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json, okf\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Log warning, continue _(configured but not yet enforced — cost_guard is wired to the config but check() is not called in the ingest path)_ |\n| `cost.hard_gate_usd` | float | `2.00` | Require explicit confirmation _(configured but not yet enforced — see above)_ |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × backup_count` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python scripts/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python scripts/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nThree independent cache layers:\n\n### Layer 1 — Embedding cache (`embeddings.db`)\n\nStores the BM25 index entry for each wiki page, keyed by page content SHA-256. When a page is updated, only that page's entry is refreshed.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**File:** `synthadoc/core/cost_guard.py`\n\nEnforces per-operation budget limits. Evaluated before every LLM call.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | Log warning; auto-continue |\n| `hard_gate_usd` | $2.00 | Prompt user `Proceed? [y/N]`; block if N; skip prompt if `auto_confirm=True` or `--yes` flag |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` at OS socket level. Not configurable. No remote access without a separate reverse proxy (which the user must explicitly set up).\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n references/\n format-notes.md ← optional; load with self.get_resource(\"format-notes.md\")\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\n\nclass MyProvider(LLMProvider):\n\n async def complete(\n self,\n messages: list[Message],\n model: str,\n temperature: float = 0.0,\n **kwargs,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n```\n\nPlace in `~/.synthadoc/providers/` or the wiki `providers/` directory. Reference by name in config:\n\n```toml\n[agents]\ndefault = { provider = \"my_provider\", model = \"my-model-id\" }\n```\n\n### Writing a hook\n\nHooks can be in any language. They receive JSON on stdin and must exit 0 on success.\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `--wiki-root `.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; a missing source file triggers `archived`. Manual transitions allow operators to override any state.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\n| From | To | Trigger | Condition |\n|---|---|---|---|\n| _(none)_ | `draft` | `ingest` | New page created by ingest |\n| `draft` | `active` | `lint` | Page passes all lint checks |\n| `draft` / `stale` / `contradicted` | `active` | `cli` / `api` | Manual activate (`synthadoc lifecycle activate`) |\n| `active` | `contradicted` | `lint` | Contradiction detected between two or more sources |\n| `active` | `stale` | `lint` | Local source: SHA-256 hash mismatch; or URL source older than `url_staleness_days` |\n| `active` / `stale` | `archived` | `lint` | Local source no longer exists on disk; or URL source returns 404/410 (opt-in) |\n| any | `archived` | `cli` / `api` | Manual archive (`synthadoc lifecycle archive`) |\n| `archived` | `draft` | `cli` / `api` | Manual restore (`synthadoc lifecycle restore`) — places page back in review queue |\n| `contradicted` | `archived` | `cli` / `api` | Manual archive after reviewing the conflict |\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: if the source path recorded in frontmatter no longer exists on disk, the page is transitioned to `archived`.\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in five machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n**`okf`** — [Open Knowledge Format v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle directory. Unlike other formats, `okf` produces a **directory tree** rather than a single file. The bundle is directly consumable by any OKF-aware agent or tool without code changes.\n\nBundle layout:\n```\n/\n index.md # OKF index — pages grouped by knowledge type\n log.md # lifecycle change history, newest first\n wiki/\n .md # one OKF concept file per wiki page\n```\n\nEach concept file carries a conformant frontmatter block:\n\n```yaml\ntype: person # from WikiPage.type; fallback \"concept\" for old pages\ntitle: Alan Turing\ndescription: Father of theoretical computer science and pioneer of the Turing machine.\nresource: https://example.com/turing-bio # omitted for local-file sources\ntags: mathematics, computation, cryptography\ntimestamp: '2026-04-22' # WikiPage.updated ?? WikiPage.created\nstatus: active # Synthadoc extension — OKF consumers tolerate unknown fields\nconfidence: high # Synthadoc extension\n```\n\nOKF conformance rules satisfied: (1) every `.md` has parseable frontmatter; (2) every frontmatter has a non-empty `type`; (3) reserved filenames follow spec structure. Synthadoc-specific fields (`status`, `confidence`) are preserved as extensions — the spec requires consumers to tolerate unknown keys.\n\n`[[wikilinks]]` in page bodies are rewritten to OKF-style relative paths (`[Title](slug.md)`) so cross-links are valid within the bundle.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. For `--format okf`, `--output` is required and must be a directory path. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`). For `--format okf`, only `all` (active + contradicted, the default) or `active` are meaningful — draft, stale, and archived pages are always excluded from OKF bundles.\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type`:\n\n| Format | Content-Type |\n|--------|-------------|\n| `llms.txt`, `llms-full.txt` | `text/plain; charset=utf-8` |\n| `graphml` | `application/xml` |\n| `json` | `application/json` |\n| `okf` | `application/json` — a JSON object mapping relative file paths to file contents (`{\"index.md\": \"...\", \"wiki/alan-turing.md\": \"...\"}`) |\n\nReturns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml / okf)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 10) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 10 # turns to include in each request (default: 10; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 10). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n"}, "30bd7db687163aadb1866a0814e40f5676fa4f48": {"t": 1785767299.490196, "n": 149263, "h": "b123870d5f62695e", "c": "# Synthadoc — Design Document\n\n**Version:** 0.8.0 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\ntype: person # OKF knowledge type — set by IngestAgent during compilation\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nresource: https://example.com/turing-bio # OKF primary source URL (URL sources only)\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`type` values** _(added in v0.9.0, OKF-required field)_: `concept` (default), `person`, `organization`, `technology`, `event`, `location`, `product`. Set automatically by IngestAgent during the analysis pass; absent on pages ingested before v0.9.0.\n\n**`resource`** _(added in v0.9.0, OKF-optional field)_: the primary source URL for pages ingested from a URL source. Absent for local file sources and pre-v0.9.0 pages.\n\n`resource` and `sources` are complementary, not duplicates:\n\n| | `resource` | `sources` |\n|---|---|---|\n| Purpose | OKF external citation — one clean URL for agents and humans to follow | Synthadoc internal provenance — full audit record per contributing file |\n| Cardinality | Single string (or absent) | Array — grows as more files are ingested into the same page |\n| Contents | URL only | File path, SHA-256 hash, byte size, ingestion timestamp |\n| Used for | OKF compatibility; agent consumption without Synthadoc-specific knowledge | Dedup, stale detection, cost audit trail |\n| Local file sources | Absent | Present (file path + hash) |\n| URL sources | Set to the source URL | Also present (URL + hash of URL string) |\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent, checks CostGuard\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the summary (not raw text) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `skip` / `flag_contradiction`)\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads summary (not full text) + BM25 candidates + `purpose.md` scope. Outputs per-page action: `create`, `update`, `flag`, `skip` |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of four formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp}` |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 0.6.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports dangling slugs — pages listed in ROUTING.md that no longer exist in the wiki (enabled only when ROUTING.md exists). **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [--wiki-root ] — generate ROUTING.md from index.md branch structure\n│ ├── validate [--wiki-root ] — report dangling slugs and cross-branch duplicates\n│ └── clean [--wiki-root ] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [--wiki-root ]\n├── candidates\n│ ├── list [--wiki-root ]\n│ ├── promote |--all [--wiki-root ]\n│ └── discard |--all [--wiki-root ]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Log warning, continue _(configured but not yet enforced — cost_guard is wired to the config but check() is not called in the ingest path)_ |\n| `cost.hard_gate_usd` | float | `2.00` | Require explicit confirmation _(configured but not yet enforced — see above)_ |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × backup_count` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python scripts/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python scripts/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nThree independent cache layers:\n\n### Layer 1 — Embedding cache (`embeddings.db`)\n\nStores the BM25 index entry for each wiki page, keyed by page content SHA-256. When a page is updated, only that page's entry is refreshed.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**File:** `synthadoc/core/cost_guard.py`\n\nEnforces per-operation budget limits. Evaluated before every LLM call.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | Log warning; auto-continue |\n| `hard_gate_usd` | $2.00 | Prompt user `Proceed? [y/N]`; block if N; skip prompt if `auto_confirm=True` or `--yes` flag |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` at OS socket level. Not configurable. No remote access without a separate reverse proxy (which the user must explicitly set up).\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n references/\n format-notes.md ← optional; load with self.get_resource(\"format-notes.md\")\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\n\nclass MyProvider(LLMProvider):\n\n async def complete(\n self,\n messages: list[Message],\n model: str,\n temperature: float = 0.0,\n **kwargs,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n```\n\nPlace in `~/.synthadoc/providers/` or the wiki `providers/` directory. Reference by name in config:\n\n```toml\n[agents]\ndefault = { provider = \"my_provider\", model = \"my-model-id\" }\n```\n\n### Writing a hook\n\nHooks can be in any language. They receive JSON on stdin and must exit 0 on success.\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `--wiki-root `.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; a missing source file triggers `archived`. Manual transitions allow operators to override any state.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\n| From | To | Trigger | Condition |\n|---|---|---|---|\n| _(none)_ | `draft` | `ingest` | New page created by ingest |\n| `draft` | `active` | `lint` | Page passes all lint checks |\n| `draft` / `stale` / `contradicted` | `active` | `cli` / `api` | Manual activate (`synthadoc lifecycle activate`) |\n| `active` | `contradicted` | `lint` | Contradiction detected between two or more sources |\n| `active` | `stale` | `lint` | Local source: SHA-256 hash mismatch; or URL source older than `url_staleness_days` |\n| `active` / `stale` | `archived` | `lint` | Local source no longer exists on disk; or URL source returns 404/410 (opt-in) |\n| any | `archived` | `cli` / `api` | Manual archive (`synthadoc lifecycle archive`) |\n| `archived` | `draft` | `cli` / `api` | Manual restore (`synthadoc lifecycle restore`) — places page back in review queue |\n| `contradicted` | `archived` | `cli` / `api` | Manual archive after reviewing the conflict |\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: if the source path recorded in frontmatter no longer exists on disk, the page is transitioned to `archived`.\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in four machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`).\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type` (`text/plain; charset=utf-8` for text formats, `application/xml` for graphml, `application/json` for json). Returns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 10) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 10 # turns to include in each request (default: 10; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 10). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n"}, "9b06fc90d14f1e5442b0014c91e724cbcabf94d7": {"t": 1785767301.1646042, "n": 147944, "h": "e04b10fa250a37b4", "c": "# Synthadoc — Design Document\n\n**Version:** 0.8.0 \n**Audience:** Product users who want to understand how the system works; developers adding features, skills, and plugins.\n\n**Document owners:** Paul Chen, William Johnason\n\n---\n\n## Table of Contents\n\n1. [Overview](#1-overview)\n2. [Core Concepts](#2-core-concepts)\n3. [System Architecture](#3-system-architecture)\n4. [Agents](#4-agents)\n5. [Skills System](#5-skills-system)\n6. [Storage](#6-storage)\n7. [HTTP API](#7-http-api)\n8. [Obsidian Plugin](#8-obsidian-plugin)\n9. [CLI](#9-cli)\n10. [Configuration](#10-configuration)\n11. [Hook System](#11-hook-system)\n12. [Cache System](#12-cache-system)\n13. [Cost Guard](#13-cost-guard)\n14. [Job Queue](#14-job-queue)\n15. [Observability and Logging](#15-observability-and-logging)\n16. [Security](#16-security)\n17. [Plugin Development Guide](#17-plugin-development-guide)\n18. [Routing](#18-routing)\n19. [Candidates Staging](#19-candidates-staging)\n20. [Context Packs](#20-context-packs)\n21. [Adversarial Review](#21-adversarial-review)\n22. [Claim-Level Provenance](#22-claim-level-provenance)\n23. [Lifecycle Machine](#23-lifecycle-machine)\n24. [Export Formats](#24-export-formats)\n25. [Streaming Query and Query Cache](#25-streaming-query-and-query-cache)\n26. [Web Chat UI and Session Management](#26-web-chat-ui-and-session-management)\n\n**Appendices**\n- [Appendix A — Release Feature Index](#appendix-a--release-feature-index)\n\n---\n\n## 1. Overview\n\nSynthadoc is a **domain-agnostic LLM knowledge compilation engine**. It reads raw source documents and uses an LLM to synthesize them into a persistent structured wiki. Knowledge is compiled at **ingest time** — not at query time. The compiled wiki lives as plain Markdown files that are readable and editable without any tool running.\n\n**Key design principles:**\n\n- **Ingest-time compilation** — synthesis, cross-referencing, and contradiction detection happen once per source, not on every query.\n- **Local-first** — all data stays on disk; the server binds only to `127.0.0.1`.\n- **Obsidian-native** — wiki pages are valid Obsidian notes with `[[wikilinks]]`, YAML frontmatter, and Dataview compatibility.\n- **Layered access** — CLI, HTTP REST API, and MCP server expose the same operations; the agent and storage logic is shared.\n- **Extensible by design** — skills (file formats) and providers (LLM backends) are loaded as plugins; no core changes needed to add either.\n\n---\n\n## 2. Core Concepts\n\n### Wiki\n\nA self-contained knowledge base rooted at a filesystem directory. Contains:\n\n```\nmy-wiki/\n wiki/ ← compiled Markdown pages\n raw_sources/ ← original source documents\n hooks/ ← wiki-specific hook scripts\n AGENTS.md ← LLM instructions for this domain\n log.md ← human-readable activity log\n .synthadoc/\n config.toml ← per-project configuration\n audit.db ← immutable audit trail\n jobs.db ← job queue\n cache.db ← LLM response cache\n embeddings.db ← BM25 + vector search index\n extracted/ ← plain-text sidecars for ingested local sources (v0.5.0)\n report.txt ← extracted text for report.pdf (or .docx, .xlsx, etc.)\n report.pdf.pagemap.json ← PDF page-boundary map (PDF sources only)\n logs/\n synthadoc.log ← rotating JSON-lines operational log\n traces.jsonl ← OpenTelemetry traces\n```\n\n### Wiki Page\n\nA Markdown file in `wiki/` with YAML frontmatter:\n\n```yaml\n---\ntitle: Alan Turing\ntags: [computer-science, cryptography, turing-test]\nstatus: active # active | contradicted | archived\nconfidence: high # high | medium | low\ncreated: '2026-04-10'\nsources:\n - file: turing-biography.pdf\n hash: sha256:abc123…\n size: 204800\n ingested: '2026-04-10'\n---\n\n# Alan Turing\n\nContent with [[wikilinks]] to related pages…\n```\n\n**`status` values:**\n\n| Value | Meaning |\n|-------|---------|\n| `draft` | Newly compiled — not yet lint-reviewed |\n| `active` | Lint-reviewed, current, trusted |\n| `contradicted` | A new source conflicts with this page; needs resolution |\n| `stale` | Source file changed since last ingest |\n| `archived` | Source removed or manually retired |\n\nNew pages are created with `status: draft`. Lint promotes them to `active` automatically when all checks pass. See [§23 Lifecycle Machine](#23-lifecycle-machine) for full transition rules.\n\n**`lint_warnings`** _(added in v0.5.0)_ — list of adversarial review findings written to frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This specific figure lacks scholarly consensus…\"\n```\n\nCleared automatically when `--no-adversarial` is passed to `lint run`.\n\n### Job\n\nEvery ingest, lint, and scheduled operation runs as a job:\n\n```\npending → in_progress → completed\n → failed\n → dead\n → skipped\npending → cancelled\n```\n\nJobs persist across server restarts. See [§14 Job Queue](#14-job-queue) for full state transition rules, retry backoff, and status descriptions.\n\n### Slug\n\nThe filename without extension, derived from the page title. ASCII-safe and CJK-aware:\n\n- Lowercase, hyphens for separators\n- Unicode accents decomposed (NFKD)\n- CJK characters (Chinese, Japanese, Korean) preserved as-is\n- Slug blacklist blocks reserved words (`wiki`, `obsidian`, `index`, `dashboard`, `wikilinks`)\n- Collisions resolved by appending `-2`, `-3`, etc.\n\n---\n\n## 3. System Architecture\n\n### Component Map\n\n![Synthadoc Architecture](png/architecture.png)\n\n### Request lifecycle (ingest via CLI)\n\n1. `synthadoc ingest report.pdf -w my-wiki`\n2. CLI posts `POST /jobs/ingest {source: \"report.pdf\"}` to `localhost:7070`\n3. HTTP server validates path, writes job to `jobs.db` with status `pending`, returns `{job_id}`\n4. Background worker picks up job within 2 seconds\n5. Orchestrator instantiates IngestAgent, checks CostGuard\n6. SkillAgent detects `.pdf`, lazy-loads `PdfSkill`, extracts text\n7. IngestAgent Step 1 — Analysis: `_analyse()` extracts entities, tags, and a 3-sentence summary (cached under key `analyse-v1`)\n8. IngestAgent Step 2 — Decision: LLM reads the summary (not raw text) + BM25-retrieved candidate pages + `purpose.md` scope, decides per-page action (`create` / `update` / `skip` / `flag_contradiction`)\n9. IngestAgent Step 3 — Write: applies actions; updates frontmatter; writes `[[wikilinks]]`; fires hooks\n10. IngestAgent Step 4 — Overview: if any pages were created or updated, regenerates `wiki/overview.md`\n11. Job transitions to `completed`; `log.md` updated; `audit.db` record written\n\n---\n\n## 4. Agents\n\nAll agents are async Python classes. They receive a job context, write results to storage, and return a summary. Agents never call each other directly — they are dispatched by the Orchestrator.\n\n### IngestAgent\n\nFive-pass pipeline:\n\n| Pass | Model | Purpose |\n|------|-------|---------|\n| 0 — Vision (optional) | Default | Extract text from image sources (`is_image=True`); requires a vision-capable provider |\n| 1 — Analysis (`_analyse()`) | Default | Extract entities, tags, and a 3-sentence summary from raw text. Result cached under key `analyse-v1` keyed by SHA-256 of the text. |\n| 2 — Candidate search | None (BM25) | Find existing wiki pages related to extracted entities |\n| 3 — Decision | Default | LLM reads summary (not full text) + BM25 candidates + `purpose.md` scope. Outputs per-page action: `create`, `update`, `flag`, `skip` |\n| 4 — Citation annotation (`_annotate_citations()`) | Default | For each page section being written, the LLM reads the section alongside the numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs. Results cached by section SHA-256. Falls back gracefully (returns original section) on any LLM or parse failure — ingest always completes. Citations are recorded in `audit.db` `claim_citations` table. |\n| 5 — Write | None | Apply actions; update frontmatter; write `[[wikilinks]]`; fire hooks. For local sources, writes a `.txt` sidecar to `.synthadoc/extracted/` (all file types) and a pagemap JSON sidecar when PDF page boundaries are available. |\n| 6 — Overview | Default | Regenerate `wiki/overview.md` if any pages were created or updated |\n\n**Analysis caching:** The analysis step is expensive (full text read + LLM call). Results are cached in `cache.db` by text SHA-256. Subsequent ingests of the same source (e.g. after a `--force` that hits the decision cache miss) re-use the analysis result without a new LLM call.\n\n**purpose.md scope filtering:** IngestAgent reads `wiki/purpose.md` at init. Its content is prepended to the decision prompt. The LLM can respond with `action=\"skip\"` when the source is clearly outside the wiki's stated scope. If `purpose.md` is absent, all sources are accepted.\n\n**overview.md auto-maintenance:** After any ingest that creates or updates pages, IngestAgent calls `_update_overview()`, which reads the 10 most-recently-modified wiki pages and asks the LLM to write a 2-paragraph overview of the entire wiki. The result is saved to `wiki/overview.md` with `status: auto` frontmatter. This page is excluded from contradiction detection and orphan checks.\n\n**Web search fan-out:** When a source is routed to the `web_search` skill, `ExtractedContent.metadata[\"child_sources\"]` contains the top result URLs. IngestAgent detects this and returns early with the URL list; the Orchestrator enqueues each URL as a separate ingest job. This keeps the web search skill stateless and the queue the single source of work.\n\n**Deduplication:** Every source tracked by SHA-256 in `audit.db`. Hash match → skip. Use `--force` to bypass.\n\n**Slug derivation:**\n\n```python\ndef _slugify(title: str) -> str:\n normalized = unicodedata.normalize(\"NFKD\", title)\n slug = re.sub(\n r\"[^a-z0-9\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]+\",\n \"-\", normalized.lower(),\n ).strip(\"-\")\n return slug or \"page-\" + hashlib.md5(title.encode()).hexdigest()[:8]\n```\n\n**Contradiction flagging:** When Pass 3 returns `flag_contradiction`, the page's frontmatter is updated to `status: contradicted`, both the old claim and new conflicting claim are preserved with `⚠` markers and citations.\n\n**CJK support:** Entity extraction falls back to CJK 2–6 char sequence regex when SpaCy is unavailable. `_slugify` preserves CJK characters. BM25 tokenizer handles CJK unigrams.\n\n### QueryAgent\n\n#### Query Decomposition\n\n**Pipeline:**\n\n```\nQuestion\n → Call 1: decompose() — LLM splits question into 1–N sub-questions (cap=4)\n └─ on any LLM error: fall back to [question] graceful degradation\n → parallel BM25 search per sub-question asyncio.gather()\n → merge candidates — best score wins per slug deduplication\n → Call 2: LLM synthesises answer from merged context unchanged from v0.1\n → record_query() in audit.db cost + history tracking\n → log_query() in activity log operator visibility\n```\n\n**Decomposition behaviour:**\n- Simple questions decompose to a single sub-question — identical behaviour to v0.1\n- Compound questions (e.g. \"Who invented FORTRAN and what was the Bombe machine?\") decompose into one sub-question per part — each part retrieved independently, pages merged before synthesis\n- Comparative questions (e.g. \"Compare Turing's contributions with Von Neumann's\") retrieve both subjects in parallel\n- The LLM returns a JSON array of strings. Markdown code fences (` ```json ``` `) are stripped before parsing — required for cross-model robustness (some providers wrap JSON in fences despite instructions)\n- On any failure during decomposition (network error, invalid JSON, empty list, non-array response), the agent falls back silently to `[question]` — the query always completes\n\n**BM25 corpus cache:** `HybridSearch` builds the BM25 corpus once per server session and caches it in memory (`_cached_corpus`). The cache is invalidated by `invalidate_index()` after every `write_page()` call in IngestAgent, so queries always see current wiki content without redundant disk reads.\n\n#### Knowledge Gap Workflow\n\nAfter the BM25 merge step, a knowledge gap is detected when ANY of three independent signals fire (gap is skipped when `gap_score_threshold = 0`):\n\n1. `len(candidates) < 3` — wiki has almost nothing on the topic\n2. `max_score < gap_score_threshold` (default: `2.0`, configurable via `[query] gap_score_threshold` in `config.toml`) — low keyword overlap\n3. Fewer than 2 candidates contain any key noun from the question with sufficient frequency — corpus-relative BM25 scores can be inflated by shared vocabulary; this content-overlap check catches off-topic matches\n\nWhen a gap fires:\n\n1. `SearchDecomposeAgent.decompose(question)` is called to generate 1–4 focused keyword search strings\n2. `QueryResult.knowledge_gap = True` and `QueryResult.suggested_searches = [...]` are set\n3. The CLI appends a `[!tip] Knowledge Gap Detected` Obsidian callout with:\n - Obsidian Command Palette path (primary)\n - `synthadoc ingest \"search for: ...\"` terminal commands (with `-w`)\n4. The API response includes `knowledge_gap` and `suggested_searches` fields\n5. The Obsidian `QueryModal` renders the same callout using `MarkdownRenderer.render()`\n\nWhen no gap is detected, `suggested_searches` is `[]` and no callout is shown.\n\n---\n\n### Web Search Decomposition (v0.2.0)\n\n> **Note:** Implementation is in `docs/plans/web-search-decomposition-v0.2.md`. This section describes the delivered behavior.\n\n**Motivation:** The v0.1 web search feature (`synthadoc ingest \"search for: \"`) fired a single Tavily API call for the entire input phrase. Decomposing the search intent into multiple focused keyword queries before fetching produces richer, more targeted pages — each sub-query targets a different aspect of the topic.\n\n**Pipeline:**\n\n```\nUser input: \"search for: yard gardening in Canadian climate zones\"\n → IngestAgent detects web_search skill\n → strip intent prefix → \"yard gardening in Canadian climate zones\"\n → SearchDecomposeAgent.decompose() — LLM returns terse keyword strings\n e.g. [\"Canada hardiness zones map\",\n \"planting guide by province Canada\",\n \"frost dates Canadian cities\"]\n → asyncio.gather() — N parallel Tavily API calls\n → deduplicate URLs across results (first-seen wins, order preserved)\n → merged child_sources → existing fan-out unchanged\n```\n\n**Key design decisions:**\n- Uses a **separate prompt** from `QueryAgent.decompose()` — query decomposition asks \"what distinct *questions* does this ask?\" (natural-language sub-questions) while search decomposition asks \"what distinct *search strings* would find the best authoritative sources?\" (terse keyword phrases). The outputs are fundamentally different — they must not share a prompt.\n- Implemented as `SearchDecomposeAgent` in `synthadoc/agents/search_decompose_agent.py` — kept separate to avoid coupling the two decomposition strategies.\n- Cap: 4 search strings maximum — prevents runaway Tavily API spend.\n- Fallback: if LLM call fails, JSON is invalid, or all entries are whitespace, use the original phrase as a single search query — the ingest always completes.\n\n### Semantic Re-ranking\n\n> **Opt-in.** BM25 is the default and works without any additional dependencies.\n\n**Installation:**\n\n```bash\npip install fastembed\n```\n\n**Enable in config:**\n\n```toml\n[search]\nvector = true\nvector_top_candidates = 20 # BM25 candidate pool; top_n returned after re-ranking\n```\n\n**Embedding model:** `BAAI/bge-small-en-v1.5` (~130 MB), managed by `fastembed`. Downloaded once on the first server start with `vector = true`; cached at `~/.cache/fastembed/` thereafter.\n\n**On first enable**, the server prints and logs:\n\n```\nVector search enabled — downloading embedding model BAAI/bge-small-en-v1.5 (~130 MB)\nto ~/.cache/fastembed/. This is a one-time download.\n```\n\n**Search flow (when `vector = true`):**\n\n1. BM25 retrieves top `vector_top_candidates` (default 20) candidates\n2. The query is embedded; cosine similarity is computed against each candidate's stored vector\n3. Results are re-ranked by vector score; top `top_n` (default 8) are returned to the caller\n\n**Migration:** On first enable, a background task embeds all existing wiki pages into `embeddings.db`. BM25 continues to serve all queries during migration — no downtime. Progress is logged every 50 pages. New pages are embedded immediately on write.\n\n**Fallback:** If `embeddings.db` is empty, the model is unavailable, or `fastembed` is not installed, BM25 ranking is used automatically with no error.\n\n**Performance notes:**\n- First enable on a large wiki may take several minutes to embed all pages. Subsequent server starts are instant (model and embeddings already cached).\n- The re-ranking step is CPU-only and adds single-digit milliseconds per query after migration.\n- Set `vector = false` to revert to BM25-only at any time. Existing embeddings are not deleted.\n\n---\n\n### LintAgent\n\nRuns against the entire wiki or a scoped subset:\n\n| Check | What it finds |\n|-------|---------------|\n| Contradiction | Pages with `status: contradicted` |\n| Orphan | Pages with zero inbound `[[wikilinks]]` |\n| Stale | Pages whose `sources[]` entries no longer exist on disk |\n| Missing link | Entity mentioned in page body but no wikilink created |\n| Adversarial review _(v0.5.0)_ | Independent LLM pass that flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support |\n| Lifecycle — archived detection _(v0.6.0)_ | Source file no longer on disk → transition page to `archived` |\n| Lifecycle — stale detection _(v0.6.0)_ | SHA-256 hash of source on disk ≠ recorded ingest hash → transition page to `stale` |\n| Lifecycle — draft promotion _(v0.6.0)_ | `draft` page with no active issues → transition to `active` |\n| Lifecycle — manual-edit sync _(v0.6.0)_ | Frontmatter `status` differs from `page_states` DB record → reconcile DB to match |\n\n**Auto-resolution:** For contradictions, LintAgent asks the LLM to propose a resolution with a confidence score. If score ≥ `auto_resolve_confidence_threshold` (default 0.85), applies automatically. Below threshold, queues for human review.\n\n**Index suggestion:** For orphan pages, LintAgent reads the page frontmatter and generates a ready-to-paste `wiki/index.md` entry: `- [[slug]] — tag1, tag2, tag3`.\n\n**Orphan frontmatter sync:** After computing orphans, both `LintAgent.lint()` (server-side, via `POST /jobs/lint`) and `synthadoc lint report` (CLI, offline) write `orphan: true` or `orphan: false` to each eligible page's YAML frontmatter. This keeps the Obsidian Dataview query (`WHERE orphan = true`) in sync with the computed orphan state without requiring the server to be running after `lint report`.\n\n**Auto-generated page exclusions:** The pages `index`, `dashboard`, `overview`, `log`, and `purpose` are excluded from both orphan detection and contradiction checking. Links from these pages do not count as real inbound references — a page linked only from `overview.md` is still reported as an orphan. These pages are also never flagged as contradicted by the ingest pipeline.\n\n**Adversarial review _(v0.5.0)_:** After structural checks complete, `LintAgent` runs an independent LLM review of every non-excluded page concurrently via `asyncio.gather()` — a 100-page wiki completes in wall-clock time equal to one call. The adversarial provider is configured via `[agents].adversarial` (falls back to `[agents].default` if absent); using a different model family from the ingest model reduces self-serving bias. Results are stored as `lint_warnings: [{claim, concern}]` in each page's YAML frontmatter. The cap is `adversarial_max_per_page` (default 2). Rate-limit failures are caught per-page and stored as non-fatal entries. Skipped entirely when `--no-adversarial` is passed to `lint run`; in that case, existing `lint_warnings` are cleared from all pages.\n\n### SkillAgent\n\nDispatches to the correct skill based on file extension, URL prefix, or intent keyword match. Manages 3-tier lazy loading. Returns `ExtractedContent` to IngestAgent.\n\nWhen a source is a URL or an intent phrase (e.g. `search for: Dennis Ritchie`), IngestAgent skips the local file checks — there is no file to verify or hash. File-existence validation and SHA-256 dedup only apply to local file paths.\n\n### ExportAgent\n\nSerialises the wiki to one of four formats with zero additional LLM calls. Invoked via `synthadoc export --format ` or the Obsidian **Export Wiki** command.\n\n| Format | Output |\n|--------|--------|\n| `llms.txt` | Active pages as a compact index (title + first-line summary) in the [llmstxt.org](https://llmstxt.org) spec; pages with `contradicted` or `stale` status appear in a **Needs Review** section |\n| `llms-full.txt` | Full page content for all pages, separated by `---` dividers with status and confidence headers; no size limit |\n| `graphml` | Directed wikilink graph — one node per page, one edge per `[[wikilink]]`; includes `label` (Gephi), `y:NodeLabel` (yEd), status, confidence, orphan flag, inbound link count, and routing branch per node |\n| `json` | Full structured dump: content, tags, sources, claims (from audit DB), lifecycle history, routing branch memberships, per-page `ingest_cost_usd` and `ingest_tokens`, and total compilation cost |\n\n**Status filter:** All formats accept `--status-filter active|contradicted|stale|archived|draft|all` (default `all`) to scope the export to a lifecycle subset.\n\n**GraphML tool compatibility:** The file includes both a standard `label` data key (read by Gephi and Cytoscape) and a `y:ShapeNode/y:NodeLabel` element (read by yEd). No position data is embedded — run the tool's own layout algorithm after import.\n\n---\n\n## 5. Skills System\n\nSkills extract text from source documents. They are Python classes that subclass `BaseSkill` (`synthadoc/skills/base.py`, Apache-2.0).\n\n### Folder-based skill structure\n\nEach skill is a self-contained directory:\n\n```\npdf/\n SKILL.md ← YAML frontmatter (parsed by engine) + Markdown body (for humans/LLMs)\n scripts/\n main.py ← BaseSkill subclass; entry point declared in SKILL.md\n assets/ ← data files bundled with the skill (optional)\n references/ ← reference documents loaded via get_resource() (optional)\n```\n\n**`SKILL.md` frontmatter schema:**\n\n```yaml\nname: pdf\nversion: \"1.0\"\ndescription: Extract text from PDF documents\nentry:\n script: scripts/main.py\n class: PdfSkill\ntriggers:\n extensions: [\".pdf\"]\n intents: [\"pdf\", \"research paper\", \"document\"]\nrequires: [pypdf, pdfminer.six]\n```\n\nThe Markdown body is for human readers and LLMs — never engine-parsed. Use it to document usage, edge cases, and references.\n\n### 3-Tier Lazy Loading\n\n| Tier | What loads | When |\n|------|-----------|------|\n| 1 — Metadata | `SkillMeta` parsed from `SKILL.md` frontmatter | Always; startup |\n| 2 — Body | Full skill class via `importlib.util` | When a matching source is encountered |\n| 3 — Resources | Files from `assets/` or `references/` via `get_resource()` | On first access within the skill |\n\nThis means importing 20 skills costs essentially zero memory until they are needed.\n\n### Registry cache\n\n`SkillAgent` writes `skill_registry.json` to `/.synthadoc/` on init. Each entry stores the `SKILL.md` mtime; on subsequent startups, unchanged entries are deserialised without re-parsing YAML (warm start). New, changed, or deleted skill folders are detected automatically.\n\n### Intent-based dispatch\n\n`detect_skill(source)` matches against `triggers.extensions` (file suffix or URL prefix) **and** `triggers.intents` (substring match on lowercased source string). This enables purely intent-driven skills with no file extension — e.g., `web_search` triggers on `\"search for\"`, `\"look up\"`, `\"find on the web\"`, etc.\n\nFor URL sources, **longest prefix wins**: the matched extension string length determines priority. A skill with prefix `https://www.youtube.com/` (28 chars) takes priority over the generic URL skill prefix `https://` (8 chars). This makes Tavily web search results that happen to be YouTube links automatically routed to the YouTube skill without any special-case logic.\n\n### Built-in Skills\n\n| Skill | Extensions | Intent phrases | Notes |\n|-------|-----------|---------------|-------|\n| `pdf` | `.pdf` | `pdf`, `research paper`, `document` | pypdf primary; pdfminer.six fallback if yield < 50 chars/page |\n| `url` | `http://`, `https://` | `fetch url`, `web page`, `website` | httpx fetch + BeautifulSoup clean |\n| `markdown` | `.md`, `.txt` | `markdown`, `text file`, `notes` | Direct read |\n| `docx` | `.docx` | `word document`, `docx` | python-docx |\n| `pptx` | `.pptx` | `powerpoint`, `presentation`, `pptx` | python-pptx; each slide rendered as a titled section; speaker notes appended when present |\n| `xlsx` | `.xlsx`, `.csv` | `spreadsheet`, `excel`, `csv` | openpyxl |\n| `image` | `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff` | `image`, `screenshot`, `diagram`, `photo` | Base64 + vision LLM |\n| `web_search` | _(none)_ | `search for`, `find on the web`, `look up`, `web search`, `browse` | Calls Tavily API; returns top result URLs as child sources enqueued individually. Requires `TAVILY_API_KEY`. |\n| `youtube` | `https://www.youtube.com/`, `https://youtu.be/` | `youtube video`, `youtube lecture`, `youtube talk` | Extracts captions via YouTube caption system; no API key or audio download needed. Generates an executive summary (what the video covers, main topics, key takeaway) followed by the full timestamped transcript. Skips gracefully when no captions are available. |\n\n### Custom Skill Locations\n\nSkills are discovered from five locations in priority order:\n\n| Source | Path | Override priority |\n|--------|------|------------------|\n| `extra_dirs` (programmatic) | Passed at `SkillAgent()` init | Highest |\n| Local wiki | `/skills/` | High |\n| Global user | `~/.synthadoc/skills/` | Medium |\n| pip entry points | `entry_points('synthadoc.skills')` | Low |\n| Built-in | Ships with package (`synthadoc/skills/`) | Lowest |\n\nNo server restart needed — registry cache detects changes automatically on next startup.\n\n### BaseSkill Interface\n\n```python\n# synthadoc/skills/base.py (Apache-2.0)\n@dataclass\nclass Triggers:\n extensions: list[str] # e.g. [\".pdf\"] or [\"http://\", \"https://\"]\n intents: list[str] # e.g. [\"search for\", \"look up\"]\n\n@dataclass\nclass SkillMeta:\n name: str\n description: str\n version: str\n entry_script: str # relative path within skill_dir\n entry_class: str # class name in that script\n triggers: Triggers\n requires: list[str] # pip distribution names\n skill_dir: Path = None # set by SkillAgent after loading\n\n@dataclass\nclass ExtractedContent:\n text: str\n source_path: str\n metadata: dict = field(default_factory=dict)\n\nclass BaseSkill(ABC):\n\n @abstractmethod\n async def extract(self, source: str) -> ExtractedContent: ...\n\n def get_resource(self, filename: str) -> str:\n \"\"\"Load a file from assets/ or references/ within the skill folder.\"\"\"\n ...\n```\n\n---\n\n## 6. Storage\n\n### wiki/ — Page files\n\nPlain Markdown. One file per page. Filename = slug + `.md`. Frontmatter is YAML between `---` delimiters. Body uses standard Markdown with `[[wikilinks]]` for internal references.\n\n### audit.db — Immutable audit trail\n\nSQLite. Two key tables:\n\n**`ingest_log`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `source` | TEXT | Original path or URL |\n| `hash` | TEXT | `sha256:` |\n| `size` | INTEGER | Bytes |\n| `cost_usd` | REAL | |\n| `tokens` | INTEGER | |\n| `pages_created` | TEXT | JSON array of slugs |\n| `pages_updated` | TEXT | JSON array of slugs |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`audit_events`**\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `event` | TEXT | e.g. `contradiction_found`, `auto_resolved`, `cost_gate_triggered` |\n| `details` | TEXT | JSON |\n| `recorded_at` | TEXT | UTC ISO-8601 |\n\n**`queries`** _(added in v0.2.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `question` | TEXT | Original question text |\n| `sub_questions_count` | INTEGER | Number of sub-questions decomposed (1 for simple questions) |\n| `tokens` | INTEGER | Answer call token usage |\n| `cost_usd` | REAL | Approximate cost (answer tokens × rate) |\n| `queried_at` | TEXT | UTC ISO-8601 |\n\n**`claim_citations`** _(added in v0.5.0)_\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Filename of the raw source (basename only) |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph (for display) |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n**`page_states`** _(added in v0.6.0)_\n\nFast slug-keyed current state index. One row per wiki page.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `slug` | TEXT PK | Wiki page slug |\n| `state` | TEXT | One of: `draft`, `active`, `contradicted`, `stale`, `archived` |\n| `updated_at` | TEXT | UTC ISO-8601 — when this row was last modified |\n| `triggered_by` | TEXT | Who caused the last transition: `ingest`, `lint`, `cli`, `api` |\n\n**`lifecycle_events`** _(added in v0.6.0)_\n\nImmutable append-only audit log of every lifecycle transition.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `slug` | TEXT | Wiki page slug |\n| `from_state` | TEXT | Previous state (`null` on first creation) |\n| `to_state` | TEXT | New state |\n| `reason` | TEXT | Human-readable reason (empty string if none provided) |\n| `triggered_by` | TEXT | `ingest`, `lint`, `cli`, `api` |\n| `timestamp` | TEXT | UTC ISO-8601 |\n\n### jobs.db — Job queue\n\nSee [Section 14 — Job Queue](#14-job-queue).\n\n### cache.db — LLM response cache\n\nSee [Section 12 — Cache System](#12-cache-system).\n\n### embeddings.db — Search index\n\nBM25 + optional vector index over all wiki pages. When vector search is disabled (default), only the BM25 index is used. When `[search] vector = true`, the same SQLite file also stores a `embeddings` table holding `float32` embedding vectors alongside the BM25 entries.\n\n**BM25 tokenizer** handles ASCII and CJK:\n\n```python\n@staticmethod\ndef _tokenize(text: str) -> list[str]:\n ascii_tokens = re.findall(r\"[a-z0-9]+\", text.lower())\n cjk_tokens = re.findall(\n r\"[\\u4e00-\\u9fff\\u3040-\\u309f\\u30a0-\\u30ff\\uac00-\\ud7af]\", text\n )\n return ascii_tokens + cjk_tokens\n```\n\nNote: BM25 IDF requires a minimum of 3 documents in the corpus for non-zero scores when a term appears in exactly one document (formula: `log((N-df+0.5)/(df+0.5))`; N=2, df=1 → log(1) = 0).\n\n---\n\n## 7. HTTP API\n\n**Base URL:** `http://127.0.0.1:` (default port: 7070)\n\n### Middleware\n\n- **CORS:** Allows `app://obsidian.md`, `http://localhost:*`, `http://127.0.0.1:*`\n- **ContentSizeLimitMiddleware:** Rejects bodies > 10 MB with HTTP 413\n- **Asyncio semaphore:** Max 20 concurrent requests\n- **Timeout:** 60 seconds per request\n\n### Endpoints\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `POST` | `/jobs/ingest` | `{source: str}` | `{job_id: str}` |\n| `POST` | `/jobs/lint` | `{scope?: str}` | `{job_id: str}` |\n| `GET` | `/jobs` | `?status=&sort=&order=` | `[Job]` |\n| `GET` | `/jobs/{id}` | — | `Job` |\n| `DELETE` | `/jobs/{id}` | — | `{deleted: job_id}` |\n| `GET` | `/query` | `?q=` | `{answer: str, citations: [str]}` |\n| `POST` | `/query` | `{question: str, save?: bool}` | `{answer: str, citations: [str], slug?: str}` |\n| `GET` | `/status` | — | `WikiStatus` |\n| `GET` | `/lint/report` | — | `LintReport` |\n| `GET` | `/health` | — | `{status: \"ok\"}` |\n| `GET` | `/provenance/citations` _(v0.5.0)_ | `?page=&source=&broken=&limit=N&offset=N&sort=&order=` | `{total: int, citations: [CitationRow]}` |\n| `GET` | `/lifecycle/status` _(v0.6.0)_ | — | `{draft: int, active: int, contradicted: int, stale: int, archived: int}` |\n| `GET` | `/lifecycle/events` _(v0.6.0)_ | `?slug=&to_state=&limit=N&offset=N` | `{total: int, events: [LifecycleEvent]}` |\n| `POST` | `/lifecycle/transition` _(v0.6.0)_ | `{slug: str, to_state: str, reason?: str}` | `{slug, from_state, to_state, timestamp}` |\n| `GET` | `/query/stream` _(v0.7.0)_ | `?q=&no_cache=&timeout_seconds=N` _(timeout added v0.8.0)_ | SSE stream of `data: \\n\\n` events, terminated by `data: [DONE]\\n\\n` |\n| `GET` | `/app` _(v0.7.0)_ | — | Serves the React SPA (web chat UI) |\n| `GET` | `/sessions` _(v0.8.0)_ | — | `[{session_id, first_q, last_active, turn_count, questions: [str]}]` |\n| `GET` | `/sessions/{session_id}/messages` _(v0.8.0)_ | — | `[{role, content, timestamp}]` |\n\n**`GET /jobs` query parameters:**\n\n| Parameter | Values | Default | Description |\n|-----------|--------|---------|-------------|\n| `status` | `pending` \\| `in_progress` \\| `completed` \\| `failed` \\| `skipped` \\| `dead` \\| `cancelled` | _(all)_ | Filter to one status |\n| `sort` | `created_at` \\| `status` \\| `operation` | `created_at` | Column to sort by |\n| `order` | `asc` \\| `desc` | `asc` | Sort direction |\n\n**Operation types:** `ingest` (file/URL/web-search ingest jobs) and `lint` (lint pass jobs).\n\n**Job object:**\n\n```json\n{\n \"id\": \"abc123\",\n \"status\": \"completed\",\n \"operation\": \"ingest\",\n \"created_at\": \"2026-04-10T14:32:01Z\",\n \"payload\": {\"source\": \"report.pdf\"},\n \"result\": {\"pages_created\": [\"alan-turing\"], \"cost_usd\": 0.0, \"child_job_ids\": []},\n \"progress\": {\"phase\": \"found_urls\", \"total\": 5},\n \"error\": null\n}\n```\n\nThe `progress` field is updated in real time during execution (e.g. `{\"phase\": \"searching\"}` before Tavily call, `{\"phase\": \"found_urls\", \"total\": N}` after URLs are returned). It is `null` for jobs that do not emit progress. Web search jobs additionally store `child_job_ids` in `result` so callers can track the fan-out URL ingest jobs.\n\n**LintReport object:**\n\n```json\n{\n \"contradictions\": [\"grace-hopper\"],\n \"orphans\": [\"quantum-computing\"],\n \"orphan_details\": [\n {\n \"slug\": \"quantum-computing\",\n \"index_suggestion\": \"- [[quantum-computing]] — physics, computing, qubits\"\n }\n ],\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\n`adversarial_warnings` is present in v0.5.0+; it is an empty list when no warnings were found or when the adversarial pass was skipped.\n\n**CitationRow object** _(v0.5.0)_:\n\n```json\n{\n \"page_slug\": \"alan-turing\",\n \"source_file\": \"turing-enigma-decryption.pdf\",\n \"line_start\": 8,\n \"line_end\": 10,\n \"claim_excerpt\": \"## Bletchley Park and the Bombé\",\n \"ingested_at\": \"2026-05-21T14:32:01\"\n}\n```\n\n**Note on timestamps:** All `created_at` values are stored and returned as UTC. The Obsidian plugin appends `+00:00` before passing to `new Date()` to ensure correct local-time display.\n\n### Path resolution\n\n`POST /jobs/ingest` accepts:\n- Absolute path: `/home/user/docs/report.pdf`\n- Vault-relative path: `raw_sources/report.pdf` (resolved against `wiki_root`)\n- URL: `https://example.com/article`\n\n### Background worker\n\nThe HTTP server runs a background task that polls `jobs.db` every 2 seconds and dispatches pending jobs. Max 4 concurrent ingest jobs (configurable via `max_parallel_ingest`).\n\n---\n\n## 8. Obsidian Plugin\n\n**Package:** `synthadoc-obsidian` (TypeScript) \n**Location:** `obsidian-plugin/` in the repo \n**Version:** 0.6.0\n\nEach vault configures its server URL in plugin settings (default `http://127.0.0.1:7070`).\n\n**Installation:** Build with `npm run build` in `obsidian-plugin/`, then copy `main.js` and\n`manifest.json` to `/.obsidian/plugins/synthadoc/`. Enable in Settings → Community Plugins.\nReload the plugin (toggle off/on) after copying — a full Obsidian restart is not required.\n\n### Command palette\n\n| Command | Behaviour |\n|---------|-----------|\n| `Synthadoc: Ingest...` | Tabbed modal with four ingest modes. **Web search** — type a topic, set max results (1–50, default 20) and poll interval (500–10 000 ms, default 2000 ms); polls live showing phase text, pages list, and per-URL errors until all fan-out jobs settle. `Ctrl/Cmd+Enter` to submit. **From URL** — paste any URL and queue it for ingest; polls job status live. **All sources in folder** — queues every supported file in the configured raw sources folder. **Pick files** — click **Browse…** to select a folder from the OS picker, then **Scan** to list supported files; wiki sub-folder contents and common system files are excluded automatically; select files and click **Ingest selected**. |\n| `Synthadoc: Query: ask the wiki...` | Responsive modal (min 520px, 60vw, max 860px); markdown-rendered answer with citation footer; stays open when clicking elsewhere — must be closed explicitly via ✕ or Escape |\n| `Synthadoc: Lint: report` | 3-tab modal — **Contradictions**, **Orphans**, **Adversarial** _(v0.5.0)_. The Adversarial tab shows each flagged claim (orange) with its concern and suggested re-ingest commands. |\n| `Synthadoc: Lint: run...` | Modal with **Auto-resolve** and **Skip adversarial review** _(v0.5.0)_ checkboxes. Queues a lint job; polls progress live; reports contradiction, orphan, and adversarial warning counts when complete. Tick **Skip adversarial review** to run structural-only lint (also clears existing `lint_warnings`). |\n| `Synthadoc: View Page Provenance` _(v0.5.0)_ | Sortable, paginated table of every claim citation recorded across the wiki — page, claim excerpt, source file, line range, and ingest timestamp. Draggable; all cell content is selectable and copyable. Click any row to open the Source Viewer showing the exact source lines with ±5 lines of context. For PDF sources a page-jump button opens the native PDF viewer at the correct page. |\n| `Synthadoc: Manage Page Lifecycle` _(v0.6.0)_ | Sortable, filterable, paginated table of all wiki pages with their current lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) and last transition timestamp. State filter checkboxes narrow the table; click column headers to sort. Each row shows valid transition buttons — click a button to trigger a transition; a reason dialog appears before committing. Clicking a draft or stale badge on the lint modal or jobs panel opens this table pre-filtered to that state. |\n| `Synthadoc: Jobs...` | Modal with status-filter checkboxes (pending, in_progress, completed, failed, skipped, dead, cancelled), sortable results table (click **Status**, **Operation**, or **Created** headers to sort ascending; click again to reverse; ▲/▼ indicates active sort, ⇅ indicates unsorted; default: newest first), error detail rows for failed/dead/cancelled jobs, pagination (25 per page), auto-refresh countdown, a **Retry selected** button (enabled when ≥ 1 selected job is failed/dead/cancelled) and a **Delete selected** button (enabled when ≥ 1 job is selected). A **Purge old jobs** footer row lets you set a day threshold and remove old completed/dead jobs in one click. |\n| `Synthadoc: Routing: manage ROUTING.md...` | Modal panel with three buttons. **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist). **Validate** reports dangling slugs — pages listed in ROUTING.md that no longer exist in the wiki (enabled only when ROUTING.md exists). **Clean** removes dangling slugs from ROUTING.md (enabled only when ROUTING.md exists). After each action the result appears inline with per-entry `[Branch] [[slug]]` detail rows. |\n| `Synthadoc: Staging: manage staging policy...` | Modal panel showing the current policy state. A segmented control switches between **Off**, **All**, and **Threshold**. When **Threshold** is selected, a second segmented control sets the minimum confidence (**High** / **Medium** / **Low**). A **Save** button persists the change via the HTTP API and updates the inline status. A footer link opens the Candidates modal directly. |\n| `Synthadoc: Candidates: review candidate pages...` | Paginated table (50 per page) of all staged candidate pages. Each row shows the slug, a colour-coded confidence badge, and a checkbox. **Promote All** and **Discard All** act on every candidate; **Promote Selected** and **Discard Selected** act on checked rows. The table reloads automatically after each action. A footer link opens the Staging policy modal. |\n| `Synthadoc: Context: build context pack...` | Modal with a goal/question text area, a token budget field (default 4000), and a **Build Context Pack** button (`Ctrl/Cmd+Enter` also triggers). The server decomposes the goal, retrieves and ranks wiki pages via BM25, and packs them within the budget. The result is rendered as cited Markdown in a read-only text area. **Copy to Clipboard** copies the content to the OS clipboard. **Save as .md** downloads the Markdown file with a slug-derived filename. |\n| `Synthadoc: Audit...` | Tabbed modal with four audit views, each loading automatically on open. **Query history** — last N query records (default 50) with question, sub-question count, token use, cost, and timestamp. **Ingest history** — last N ingest records (default 50) with source filename, wiki page slug, tokens, cost, and ingested-at timestamp. **Events** — last N raw audit events (default 100, max 1000) with timestamp, job ID, event type, and metadata; scrollable when tall. **Cost summary** — total tokens and cost over the last N days (default 30) plus per-day breakdown. |\n\n### Ribbon icon\n\nThe Synthadoc ribbon icon (a book icon — `synthadoc-ribbon-icon`) appears in the **left sidebar ribbon** of Obsidian, alongside other plugin icons. Click it to open the engine status at a glance.\n\nShows engine health and live page count: `✅ online · 12 pages` or `❌ offline — run 'synthadoc serve'`.\nCalls `GET /health` and `GET /status` in parallel (`Promise.allSettled`).\n\nIf the icon is not visible, make sure the plugin is enabled under **Settings → Community plugins** and that you are looking at the left ribbon (not the right sidebar). You can also pin it via right-clicking the ribbon area.\n\n### Settings\n\n| Setting | Default | Description |\n|---------|---------|-------------|\n| Server URL | `http://127.0.0.1:7070` | HTTP server for this vault |\n| Raw sources folder | `raw_sources` | Folder scanned by \"Ingest all sources\" |\n\n### Supported ingest formats\n\n`.md`, `.txt`, `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.tiff`\n\n---\n\n## 9. CLI\n\n\nThe CLI is a thin HTTP client — it posts jobs to the running server and polls for results. No LLM agents run in the CLI process.\n\n**File:** `synthadoc/cli/main.py` + subcommands in `synthadoc/cli/`\n\n### Command tree\n\n```\nsynthadoc\n├── install --target [--demo] [--domain ] [--port ]\n├── uninstall \n├── scaffold [-w wiki]\n├── demo list\n├── plugin\n│ ├── install — copy plugin files into /.obsidian/plugins/synthadoc/\n│ └── upgrade — upgrade plugin in all registered wikis at once\n├── serve [-w wiki] [--port N] [--background] [--mcp-only] [--http-only] [--verbose]\n├── ingest [-w wiki] [--batch] [--file manifest] [--force] [--analyse-only] [--max-results N]\n├── query \"\" [-w wiki] [--save] [--timeout N]\n├── lint\n│ ├── run [-w wiki] [--scope contradictions|orphans|all] [--auto-resolve] [--no-adversarial] [--no-lifecycle] [--check-urls]\n│ └── report [-w wiki]\n├── jobs\n│ ├── list [-w wiki] [--status pending|in_progress|completed|failed|skipped|dead|cancelled] [--sort created_at|status|operation] [--order asc|desc]\n│ ├── status [-w wiki]\n│ ├── retry [-w wiki]\n│ ├── delete [-w wiki]\n│ ├── cancel [-w wiki] [--yes]\n│ └── purge --older-than [-w wiki]\n├── routing\n│ ├── init [--wiki-root ] — generate ROUTING.md from index.md branch structure\n│ ├── validate [--wiki-root ] — report dangling slugs and cross-branch duplicates\n│ └── clean [--wiki-root ] — remove dangling slugs from ROUTING.md\n├── staging\n│ └── policy [off|all|threshold] [--min-confidence high|medium|low] [--wiki-root ]\n├── candidates\n│ ├── list [--wiki-root ]\n│ ├── promote |--all [--wiki-root ]\n│ └── discard |--all [--wiki-root ]\n├── context\n│ └── build \"\" [-w wiki] [--tokens N] [--output ]\n├── export -f [-o ] [-s ] [-w wiki] — llms.txt, llms-full.txt, graphml, json\n├── status [-w wiki]\n├── lifecycle\n│ ├── activate [-w wiki] [--reason \"\"]\n│ ├── archive [-w wiki] [--reason \"\"]\n│ ├── restore [-w wiki] [--reason \"\"]\n│ └── log [slug] [-w wiki] [--state ]\n├── audit\n│ ├── history [-w wiki] [--limit N] [--json]\n│ ├── cost [-w wiki] [--days N] [--json]\n│ ├── queries [-w wiki] [--limit N] [--json]\n│ ├── events [-w wiki] [--limit N] [--json]\n│ ├── citations [-w wiki] [--page ] [--source ] [--broken] [--json]\n│ └── lifecycle\n│ └── purge -w wiki (--before | --keep-latest )\n├── cache clear [-w wiki]\n└── schedule\n ├── add --op \"\" --cron \"\" [-w wiki]\n ├── list [-w wiki]\n ├── remove [-w wiki]\n ├── apply [-w wiki]\n ├── run --op \"\" [-w wiki]\n └── history [-w wiki] [-n N]\n```\n\n`synthadoc status -w ` now shows a per-state page count breakdown alongside the existing page total and job counts.\n\n### `schedule` sub-commands\n\n| Command | Description |\n|---|---|\n| `schedule add --op \"\" --cron \"\"` | Register a single recurring job with the scheduler |\n| `schedule apply` | Bulk-register all jobs declared in `[[schedule.jobs]]` in `config.toml`; idempotent alternative to running `schedule add` once per job |\n| `schedule list` | List all registered jobs with their cron expression, next run time, last run time, and last result |\n| `schedule remove ` | Remove a registered job by ID |\n| `schedule run --op \"\"` | Execute an operation immediately and record the result in the audit trail |\n| `schedule history` | Show recent scheduled run history from the audit trail |\n\n`schedule apply` is the recommended setup path when jobs are declared in `config.toml`. It reads the `[[schedule.jobs]]` array and registers every entry in one command, making schedule configuration reproducible and version-controllable:\n\n```bash\n# Declare jobs in .synthadoc/config.toml\n# [[schedule.jobs]]\n# op = \"ingest --batch raw_sources/\"\n# cron = \"0 2 * * *\"\n#\n# [[schedule.jobs]]\n# op = \"lint run\"\n# cron = \"0 3 * * 0\"\n\n# Register all declared jobs at once\nsynthadoc schedule apply -w my-wiki\n```\n\n### `query` options\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--save` | off | Save the answer as a new wiki page |\n| `--no-stream` | off | Disable token-by-token streaming; print the full answer when complete. Use in scripts, pipes, or terminals that do not handle ANSI escape codes. |\n| `--no-cache` | off | Bypass the query result cache and always call the LLM. |\n| `--timeout N` | `60` | Seconds to wait for the LLM response. Increase for slower providers (e.g. `--timeout 120` for MiniMax reasoning models) |\n\n### `ingest --analyse-only`\n\nRuns the analysis step only (entity extraction + tagging + summary) and prints the JSON result without writing any wiki pages. Useful for previewing how a source will be interpreted before committing it to the wiki.\n\n`--analyse-only` works with all three ingest modes — single source, `--batch`, and `--file` manifest. Each source is analysed in turn and its result printed as JSON:\n\n```bash\n# Single file\nsynthadoc ingest report.pdf --analyse-only -w my-wiki\n# → {\"entities\": [\"Alan Turing\", \"Enigma\"], \"tags\": [\"cryptography\"], \"summary\": \"…\"}\n\n# Whole folder — analyses every supported file, no pages written\nsynthadoc ingest --batch raw_sources/ --analyse-only -w my-wiki\n\n# Manifest — analyses each line in the file\nsynthadoc ingest --file sources.txt --analyse-only -w my-wiki\n```\n\n### `audit` sub-commands\n\nQuery the append-only `audit.db` directly from the CLI:\n\n```bash\n# Last 20 ingest records\nsynthadoc audit history -w my-wiki\n\n# Token spend + cost for the last 30 days (default) or custom window\nsynthadoc audit cost -w my-wiki\nsynthadoc audit cost --days 7 -w my-wiki\n\n# Last 100 audit events (contradictions found, auto-resolutions, cost gate triggers)\nsynthadoc audit events -w my-wiki\n```\n\n### Wiki targeting\n\nThe `-w` / `--wiki` option accepts either a **registry name** (registered via `install`) or a **filesystem path**. Without `-w`, defaults to the current working directory.\n\nRegistry stored at `~/.synthadoc/wikis.json`:\n\n```json\n{\n \"my-wiki\": \"/home/user/wikis/my-wiki\",\n \"research\": \"/home/user/wikis/research\"\n}\n```\n\n### Wiki context resolution\n\nEvery CLI command resolves the target wiki through a priority chain rather than\nrequiring `-w` on each invocation:\n\n1. **Explicit `-w `** — highest priority, always wins\n2. **`SYNTHADOC_WIKI` environment variable** — shell-session scope\n3. **`~/.synthadoc/default_wiki`** — persistent default, set by `synthadoc use `\n4. **Current directory fallback** — if `.synthadoc/config.toml` is present in CWD\n (backward compat for users who `cd` into a wiki directory)\n5. **Error** — actionable message directing user to `synthadoc use`\n\nAll hint and notification messages are written to **stderr**. Stdout carries only\ncommand results, keeping `synthadoc ... | jq` and other pipelines clean.\n\nThe `synthadoc use` command manages the saved default. `synthadoc use` (no args)\nshows which wiki is active and from which source, equivalent to `kubectl config current-context`.\n\n### Error codes\n\nEvery user-facing error carries a stable code in the format `[ERR--]`. Codes are printed to stderr and embedded in job `error` fields, making them greppable in logs.\n\n**File:** `synthadoc/errors.py`\n\n| Code | Meaning |\n|------|---------|\n| `ERR-SRV-001` | No server listening for the requested wiki |\n| `ERR-SRV-002` | Port already bound by another process |\n| `ERR-SRV-003` | Server returned a 4xx/5xx HTTP response |\n| `ERR-SRV-004` | Background server process exited immediately |\n| `ERR-WIKI-001` | Wiki root directory does not exist |\n| `ERR-WIKI-002` | Directory exists but missing `wiki/` subfolder |\n| `ERR-WIKI-003` | `wiki/` directory is not writable |\n| `ERR-WIKI-004` | Install target already exists on disk |\n| `ERR-WIKI-005` | Unknown demo template name |\n| `ERR-WIKI-006` | Name not in `~/.synthadoc/wikis.json` |\n| `ERR-CFG-001` | Required API key environment variable not set |\n| `ERR-CFG-002` | Provider name not recognised |\n| `ERR-SKILL-001` | No skill matched the source string |\n| `ERR-SKILL-002` | Required pip package for skill not installed |\n| `ERR-SKILL-003` | URL returned 403 (bot/paywall protection) |\n| `ERR-SKILL-004` | `TAVILY_API_KEY` not set for web search |\n| `ERR-INGEST-001` | Source file or directory not found |\n| `ERR-INGEST-002` | Source file exists but is empty |\n| `ERR-INGEST-003` | `--batch` target is not a directory |\n| `ERR-JOB-001` | Job ID does not exist in `jobs.db` |\n\n**CLI errors** go through the `cli_error(code, message, hint)` helper, which prints `[ERR-XXX-NNN] message` to stderr with an optional hint line and exits with code 1. **Agent and skill errors** embed the code directly in the exception message string so it surfaces in the job `error` field.\n\n---\n\n## 10. Configuration\n\n### Resolution order\n\n```\nPer-agent override → [agents].default (project) → [agents].default (global) → error\n```\n\nProject config wins over global config. Unspecified keys inherit from global defaults.\n\n### Global config — `~/.synthadoc/config.toml`\n\n```toml\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\n\n[wikis]\nresearch = \"~/wikis/research\"\n\n[observability]\nexporter = \"file\" # or \"otlp\"\notlp_endpoint = \"http://localhost:4317\" # used when exporter = \"otlp\"\n```\n\n### Provider switching\n\nAll eight supported providers (`anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`) share the same config key. Gemini, Groq, MiniMax, DeepSeek, and Qwen (DashScope) use OpenAI-compatible endpoints internally, so no custom provider class is needed — just set the provider name and supply the corresponding API key:\n\n```toml\n# Switch from Claude to Gemini Flash (free tier available)\n[agents]\ndefault = { provider = \"gemini\", model = \"gemini-2.5-flash\" }\n```\n\nRequired environment variables per provider:\n\n| Provider | Env var | Free tier | Vision |\n|----------|---------|-----------|--------|\n| `anthropic` | `ANTHROPIC_API_KEY` | No (pay-per-token) | Yes |\n| `openai` | `OPENAI_API_KEY` | No (pay-per-token) | Yes |\n| `gemini` | `GEMINI_API_KEY` | **Yes** — 15 RPM / 1M tokens/day on Flash | Yes |\n| `groq` | `GROQ_API_KEY` | **Yes** — generous free tier on Llama/Mixtral models | No |\n| `minimax` | `MINIMAX_API_KEY` | No (pay-per-token) | Yes (M2.5 / M2.7 natively multimodal) |\n| `deepseek` | `DEEPSEEK_API_KEY` | No (pay-per-token, very cheap) | No (text-only) |\n| `qwen` | `QWEN_API_KEY` | Yes — 1M free tokens (90-day trial), then paid DashScope | Model-dependent |\n| `ollama` | _(none)_ | **Yes** — fully local; **GPU required** — CPU-only inference is too slow for interactive use | Model-dependent |\n\n### Coding tool CLI providers — no API key needed\n\nIf you have an active **Claude Code** or **Opencode** subscription, you can use it as the LLM provider with no separate API key.\n\n**Requirements:** the CLI tool must be installed and reachable on `PATH`:\n- Claude Code: `claude` binary — install via `npm install -g @anthropic-ai/claude-code`\n- Opencode: `opencode` binary — install via `npm install -g opencode`\n\n**Configuration — set in `/.synthadoc/config.toml`:**\n\n```toml\n[agents]\ndefault = { provider = \"claude-code\", model = \"claude-opus-4-8\" }\nlint = { provider = \"claude-code\", model = \"claude-haiku-4-5-20251001\" }\n```\n\nFor Opencode:\n\n```toml\n[agents]\ndefault = { provider = \"opencode\", model = \"anthropic/claude-opus-4-8\" }\n```\n\n**Runtime override** — bypasses config.toml for the current server session:\n\n```bash\nsynthadoc serve -w --provider claude-code\n```\n\n**Limitations:**\n- Vector search (`search.vector = true`) is not supported — search falls back to BM25-only. Sufficient for wikis up to a few hundred pages.\n- Quota is shared with your coding tool usage. Heavy batch ingest consumes from the same daily budget. Quota exhaustion permanently fails the job (no retry) with a clear message.\n\n### Per-project config — `/.synthadoc/config.toml`\n\n```toml\n[server]\nport = 7070\n\n[agents]\ndefault = { provider = \"anthropic\", model = \"claude-opus-4-8\" }\nlint = { model = \"claude-haiku-4-5-20251001\" }\nskill = { model = \"claude-haiku-4-5-20251001\" }\n# llm_timeout_seconds = 90 # set for reasoning models to fail fast instead of silent empty response\n\n[queue]\nmax_parallel_ingest = 4\nmax_retries = 3\nbackoff_base_seconds = 5\n\n[cost]\nsoft_warn_usd = 0.50\nhard_gate_usd = 2.00\nauto_resolve_confidence_threshold = 0.85\n\n[ingest]\nmax_pages_per_ingest = 15\nchunk_size = 1500\nchunk_overlap = 150\nfetch_timeout_seconds = 30 # seconds to wait for a URL response before retrying\n\n[logs]\nlevel = \"INFO\"\nmax_file_mb = 5\nbackup_count = 5\n\n[hooks]\non_ingest_complete = \"python hooks/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python hooks/notify.py\", blocking = true } # blocking\n\n[web_search]\nprovider = \"tavily\" # only supported provider\nmax_results = 20 # URLs returned per query; each enqueued as an ingest job\n\n[audit]\nlifecycle_retention_days = 0 # 0 = keep forever (default); set to e.g. 365 to prune old events\n\n# Cron format: minute hour day-of-month month day-of-week\n# 0-59 0-23 1-31 1-12 0-6 (0=Sun)\n\n[[schedule.jobs]]\nop = \"ingest --batch raw_sources/\"\ncron = \"0 2 * * *\" # every day at 02:00\n\n[[schedule.jobs]]\nop = \"lint run\"\ncron = \"0 3 * * 0\" # every Sunday at 03:00\n```\n\n### Config keys reference\n\n| Key | Type | Default | Description |\n|-----|------|---------|-------------|\n| `wiki.domain` | str | `\"General\"` | Topic scope of the wiki. Used in prompts to keep ingest focused. Example: `\"Machine Learning\"` or `\"Quantum Computing\"`. |\n| `agents.default.provider` | str | `\"gemini\"` | LLM provider: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama` |\n| `agents.default.model` | str | `\"gemini-2.5-flash\"` | Model ID passed to the provider API |\n| `agents.default.base_url` | str | `\"\"` | Override the provider's API endpoint. Use this to point any OpenAI-compatible provider at a custom URL (e.g. a local proxy or a private deployment). |\n| `agents.default.thinking` | str | `\"\"` | Reasoning mode: `\"disabled\"` turns off chain-of-thought (faster, cheaper on MiniMax M3 and Qwen); `\"enabled\"` or `\"adaptive\"` turns it on. Empty string uses the provider's default. Applies to `minimax` and `qwen` (DashScope) providers; ignored by others. |\n| `agents.ingest.provider` | str | (inherits default) | Override provider/model for the ingest agent only. Useful to use a cheap fast model for ingest while keeping a high-quality model for queries. |\n| `agents.ingest.model` | str | (inherits default) | Model ID for the ingest agent override. |\n| `agents.query.provider` | str | (inherits default) | Override provider/model for query answering only. |\n| `agents.query.model` | str | (inherits default) | Model ID for the query agent override. |\n| `agents.lint.provider` | str | (inherits default) | Override provider/model for the lint agent only. |\n| `agents.lint.model` | str | (inherits default) | Model ID for the lint agent override. |\n| `agents.adversarial.provider` | str | (inherits default) | Dedicated LLM provider for adversarial lint review. Falls back to `agents.default` when not set. Cross-model adversarial reduces self-serving bias — a different model family evaluates claims independently. |\n| `agents.adversarial.model` | str | (inherits default) | Model ID for the adversarial reviewer. For maximum independence, choose a model from a different family than the ingest model. |\n| `agents.llm_timeout_seconds` | int | `0` | Per-call LLM timeout in seconds; `0` = no limit. Set to e.g. `90` when using reasoning models (MiniMax-M2.5, DeepSeek-R1) that can exceed their internal generation budget silently. Restart required. |\n| `agents.scaffold_max_tokens` | int | `8192` | Max output tokens for the scaffold (page generation) agent. Increase to `16384`+ when using reasoning models on large wikis where the default budget is exhausted. |\n| `agents.query_max_tokens` | int | `8192` | Max output tokens for the query agent. Increase if reasoning models exhaust their budget before completing the answer. |\n| `lint.adversarial_max_per_page` | int | `2` | Maximum adversarial warnings flagged per page. Raise to 3–5 for a thorough audit; lower to 1 to reduce noise on large wikis. |\n| `lint.check_url_availability` | bool | `false` | When `true`, lint performs an HTTP HEAD check on every URL source and flags unreachable URLs. Adds network calls to each lint run; opt-in only. |\n| `server.host` | str | `\"127.0.0.1\"` | Bind address. Change to `\"0.0.0.0\"` to expose the server on all interfaces (e.g. for LAN access). No built-in auth — restrict via firewall when exposing. |\n| `server.port` | int | `7070` | HTTP listen port. Change when running multiple wikis simultaneously. |\n| `ingest.max_pages_per_ingest` | int | `15` | Max pages one ingest job may create or update. |\n| `ingest.chunk_size` | int | `1500` | Text chunk size in characters for BM25 indexing. |\n| `ingest.chunk_overlap` | int | `150` | Overlap between consecutive chunks. |\n| `ingest.fetch_timeout_seconds` | int | `30` | Seconds to wait for a URL response before failing the fetch. |\n| `ingest.staging_policy` | str | `\"off\"` | Candidate staging gate: `\"off\"` = commit pages immediately; `\"all\"` = stage all new pages for review; `\"threshold\"` = stage only pages below `staging_confidence_min`. |\n| `ingest.staging_confidence_min` | str | `\"high\"` | Minimum confidence to auto-commit when `staging_policy = \"threshold\"`. Values: `\"high\"`, `\"medium\"`, `\"low\"`. Pages below this threshold are held as candidates. |\n| `query.gap_score_threshold` | float | `2.0` | BM25 score below which a knowledge gap is detected and `suggested_searches` are returned instead of (or alongside) an answer. Lower = more sensitive gap detection. |\n| `query.context_token_budget` | int | `4000` | Token budget for context pack assembly. Increase for richer context on complex queries; decrease if hitting prompt size limits. |\n| `queue.max_parallel_ingest` | int | `4` | Max concurrent ingest agents |\n| `queue.max_retries` | int | `3` | Retries before job → dead |\n| `queue.backoff_base_seconds` | int | `5` | Exponential backoff base (±20% jitter) |\n| `cache.version` | str | `\"4\"` | Bump to invalidate all cached LLM responses without touching source code |\n| `cost.soft_warn_usd` | float | `0.50` | Log warning, continue _(configured but not yet enforced — cost_guard is wired to the config but check() is not called in the ingest path)_ |\n| `cost.hard_gate_usd` | float | `2.00` | Require explicit confirmation _(configured but not yet enforced — see above)_ |\n| `cost.auto_resolve_confidence_threshold` | float | `0.85` | Auto-apply lint resolutions above this score |\n| `chat.conversation_history_turns` | int | `5` | Number of prior conversation turns injected into each query prompt for multi-turn context. Set to `0` to disable conversation history (each query answered independently). |\n| `chat.session_retention_days` | int | `30` | Days to retain chat session history in `audit.db`. Sessions older than this are pruned automatically. |\n| `audit.lifecycle_retention_days` | int | `0` | Days to retain lifecycle events in `audit.db`. `0` = keep forever. When set, events older than this threshold are pruned at the end of each lint run. |\n| `audit.url_staleness_days` | int | `0` | Days after which URL-sourced pages are automatically marked `stale` if the source URL has not been re-ingested. `0` = disabled. |\n| `logs.level` | str | `\"INFO\"` | Console log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |\n| `logs.max_file_mb` | int | `5` | Rotate `synthadoc.log` at this size (MB) |\n| `logs.backup_count` | int | `5` | Rotated log files to keep; total disk ≈ `max_file_mb × backup_count` |\n| `web_search.provider` | str | `\"tavily\"` | Web search provider (currently only `tavily` supported) |\n| `web_search.max_results` | int | `20` | Maximum results fetched per web search query |\n| `search.vector` | bool | `false` | Enable semantic re-ranking; downloads `BAAI/bge-small-en-v1.5` (~130 MB) once on first enable |\n| `search.vector_top_candidates` | int | `20` | BM25 candidate pool size when vector re-ranking is active |\n\n---\n\n## 11. Hook System\n\nHooks are shell commands executed when lifecycle events fire. They are configured in `.synthadoc/config.toml` under `[hooks]` and receive a JSON context object on stdin.\n\n### Configuration\n\n```toml\n# .synthadoc/config.toml\n\n[hooks]\non_ingest_complete = \"python scripts/auto_commit.py\" # non-blocking\non_lint_complete = { cmd = \"python scripts/notify.py\", blocking = true } # blocking\n```\n\n### Blocking vs. non-blocking\n\n- **Non-blocking** (default): runs in a background thread; failures are logged but do not affect the operation.\n- **Blocking**: must exit `0` for the operation to succeed; a non-zero exit code raises an error and surfaces it to the caller.\n\n### Events\n\nTwo events are fired in v0.1:\n\n| Event | Fires when | Context fields |\n|-------|-----------|----------------|\n| `on_ingest_complete` | A source is successfully ingested | `event`, `wiki`, `source`, `pages_created`, `pages_updated`, `pages_flagged`, `tokens_used`, `cost_usd` |\n| `on_lint_complete` | A lint run finishes | `event`, `wiki`, `contradictions_found`, `orphans` |\n\n### Context JSON examples\n\n**on_ingest_complete**\n```json\n{\n \"event\": \"on_ingest_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"source\": \"report.pdf\",\n \"pages_created\": [\"alan-turing\"],\n \"pages_updated\": [\"computing-history\"],\n \"pages_flagged\": [],\n \"tokens_used\": 4820,\n \"cost_usd\": 0.031\n}\n```\n\n**on_lint_complete**\n```json\n{\n \"event\": \"on_lint_complete\",\n \"wiki\": \"/home/user/wikis/my-wiki\",\n \"contradictions_found\": 2,\n \"orphans\": [\"stub-page\", \"draft-notes\"]\n}\n```\n\n### Hook library\n\nThe [`hooks/`](../hooks/) folder in the repository is a community-maintained\nlibrary of ready-to-use scripts. Copy a script to your wiki root and configure\nit in `config.toml`.\n\n**Writing a hook script:**\n\n- Read context from `sys.stdin` (JSON) — never from files or env vars\n- Write human-readable status to `sys.stderr` (not stdout)\n- Exit `0` on success, non-zero on failure\n- Include the standard header block (event, description, setup instructions)\n\nSee [`hooks/README.md`](../hooks/README.md) for contribution guidelines and\nthe full list of available scripts.\n\n---\n\n## 12. Cache System\n\nThree independent cache layers:\n\n### Layer 1 — Embedding cache (`embeddings.db`)\n\nStores the BM25 index entry for each wiki page, keyed by page content SHA-256. When a page is updated, only that page's entry is refreshed.\n\n### Layer 2 — LLM response cache (`cache.db`)\n\nStores deterministic LLM responses keyed by a hash of the operation type and full input text. Enables zero-token lint runs on unchanged pages.\n\n**Cache key:**\n\n```python\ndef make_cache_key(operation: str, inputs: dict, version: str = CACHE_VERSION) -> str:\n payload = {\"v\": version, \"op\": operation, \"inputs\": inputs}\n digest = hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()\n return digest[:32]\n```\n\nThe version is part of every cache key, so bumping it causes all existing entries to be bypassed (they remain in `cache.db` but no longer match any key).\n\nTo invalidate the cache without touching source code, set `version` in `.synthadoc/config.toml`:\n\n```toml\n[cache]\nversion = \"5\" # bump to bypass all entries cached under previous versions\n```\n\nThe default (`\"4\"`) is defined in `synthadoc/core/cache.py`. Custom skill authors and wiki operators can bump this freely without modifying core code.\n\n**Invalidation triggers:**\n\n| Trigger | Behavior |\n|---------|----------|\n| Source content changes | New SHA-256 → cache miss → fresh LLM call |\n| `[cache] version` bumped in config | All old entries bypassed |\n| `ingest --force` | `bust_cache=True` → skips `cache.get()`, repopulates |\n| `cache clear` | Deletes all rows from `cache.db` |\n\n### Layer 3 — Provider prompt cache\n\nAnthropic, OpenAI, and compatible providers cache stable prompt segments server-side. Long system prompts and `AGENTS.md` content hit this cache on repeated calls, giving 50–90% token savings.\n\n**Target cache hit rate:** > 80% on repeated lint runs across unchanged pages.\n\n---\n\n## 13. Cost Guard\n\n**File:** `synthadoc/core/cost_guard.py`\n\nEnforces per-operation budget limits. Evaluated before every LLM call.\n\n### Thresholds\n\n| Threshold | Default | Behaviour |\n|-----------|---------|-----------|\n| `soft_warn_usd` | $0.50 | Log warning; auto-continue |\n| `hard_gate_usd` | $2.00 | Prompt user `Proceed? [y/N]`; block if N; skip prompt if `auto_confirm=True` or `--yes` flag |\n\n### Cost Tracking and Pricing\n\n**How cost is computed (v0.2.0+):**\n\n```\nLLM call → CompletionResponse(input_tokens, output_tokens)\n ↓\n estimate_cost(model, input_tokens, output_tokens, is_local)\n ↓\n pricing table lookup in synthadoc/providers/pricing.py\n ↓\n IngestResult.cost_usd or audit.db queries.cost_usd\n```\n\n**Pricing table (`synthadoc/providers/pricing.py`):**\n\nA static Python dict maps model name → `(input_usd_per_token, output_usd_per_token)`.\nSeparate input and output rates reflect real-world API pricing (output tokens cost 3–5× more than input tokens for most models).\n\n| Provider | Example model | Input (per token) | Output (per token) |\n|---|---|---|---|\n| Anthropic | claude-haiku-4-5-20251001 | $0.000001 | $0.000005 |\n| Anthropic | claude-sonnet-4-6 | $0.000003 | $0.000015 |\n| OpenAI | gpt-4o-mini | $0.00000015 | $0.0000006 |\n| Gemini | gemini-2.5-flash | $0.0000003 | $0.0000025 |\n| Groq | llama-3.3-70b-versatile | $0.00000059 | $0.00000079 |\n| MiniMax | MiniMax-M2.5 | $0.00000015 | $0.0000012 |\n| MiniMax | MiniMax-M2.7 | $0.0000003 | $0.0000012 |\n\n**Special cases:**\n- **Ollama (local inference):** Always `$0.00` regardless of token count — `is_local=True` short-circuits the calculation.\n- **Unknown models:** Use a conservative fallback rate (`$0.000003` per token for both input and output) rather than crashing or silently reporting `$0.00`.\n\n**Token propagation:**\n\n- `CompletionResponse` (already in v0.1) carries `input_tokens` and `output_tokens` from every provider.\n- `QueryResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator.query()` calls `estimate_cost()` to compute `cost_usd` before writing to `audit.db`.\n- `IngestResult` gains `input_tokens` and `output_tokens` fields (v0.2.0); `Orchestrator._run_ingest()` calls `estimate_cost()` after ingest completes.\n- The vision call and analysis call in `IngestAgent` also accumulate tokens; the analysis call only has a total (split not available due to internal caching).\n\n**Refresh cadence:** The pricing table is refreshed at each major release. `_LAST_UPDATED` in `pricing.py` records the date of last review. See `CONTRIBUTING.md` for the release checklist.\n\n### API\n\n```python\nclass CostEstimate:\n tokens: int\n cost_usd: float\n operation: str\n\nclass CostGuard:\n def check(\n self,\n estimate: CostEstimate,\n auto_confirm: bool = False, # HTTP server / batch: always proceed\n interactive: bool = True, # CLI: prompt; HTTP server: False\n ) -> None: ...\n```\n\nThe HTTP server always passes `auto_confirm=True` (no interactive terminal available). The CLI passes `interactive=True`.\n\n---\n\n## 14. Job Queue\n\n**File:** `synthadoc/core/queue.py` \n**Storage:** `/.synthadoc/jobs.db` (SQLite)\n\n### State transitions\n\n```\npending → in_progress (worker picks up job)\npending → cancelled (user-initiated; `synthadoc jobs cancel`)\n\nin_progress → completed\nin_progress → failed (non-retryable error; permanent, no retry)\nin_progress → pending (retryable error; retries < max_retries, after backoff)\nin_progress → dead (retryable error; retries == max_retries)\nin_progress → skipped (system-initiated skip; e.g. auto-blocked domain)\n```\n\n| Status | Meaning | Action |\n|--------|---------|--------|\n| `failed` | Non-retryable error (e.g. stub skill, bad source) | Inspect error; fix source; enqueue again |\n| `dead` | Retryable error exhausted max retries | `synthadoc jobs retry ` to reset to pending |\n| `skipped` | System-initiated permanent skip (e.g. domain auto-blocked after repeated 403s) | No action needed; remove domain from blocked list to re-enable |\n| `cancelled` | Pending job cancelled by user via `synthadoc jobs cancel` | Re-enqueue manually if cancelled in error |\n\n**Backoff formula:** `backoff_base_seconds × 2^(retry_count) × jitter` \nwhere `jitter ∈ [0.8, 1.2]` (±20% random). Applied only to retryable errors (LLM API timeouts, 5xx responses).\n\n**Persistence:** Jobs survive server restarts. `in_progress` jobs at shutdown are reset to `pending` on startup.\n\n---\n\n## 15. Observability and Logging\n\n**Files:** `synthadoc/core/logging_config.py`, `synthadoc/observability/telemetry.py`\n\n### Handler stack\n\n```\nRoot logger (level: DEBUG)\n├── Console handler\n│ Level : cfg.logs.level (default INFO); overridden to DEBUG if --verbose\n│ Format : \"HH:MM:SS LEVEL logger — message\"\n│ Target : stderr\n│ Note : suppressed when --background spawns the detached child process\n│\n└── File handler (RotatingFileHandler)\n Level : DEBUG always\n Format : JSON lines\n Target : /.synthadoc/logs/synthadoc.log\n Rotate : cfg.logs.max_file_mb MB; cfg.logs.backup_count old files kept\n```\n\nSuppressed to WARNING: `httpx`, `httpcore`, `uvicorn.access`, `anthropic`, `openai`.\n\n**Background mode (`--background` / `-b`):** the parent process prints the startup banner, spawns a detached child process (`pythonw.exe` on Windows, `start_new_session=True` on Unix), and exits — returning the shell to the user. The child runs without a console handler; all output goes to the file handler only. PID is written to `/.synthadoc/server.pid`.\n\n### Log record fields\n\n| Field | Always present | Source |\n|-------|---------------|--------|\n| `ts` | Yes | `record.created` |\n| `level` | Yes | `record.levelname` |\n| `logger` | Yes | `record.name` |\n| `msg` | Yes | `record.getMessage()` |\n| `job_id` | Job context only | `LoggerAdapter.extra` |\n| `operation` | Job context only | `LoggerAdapter.extra` |\n| `wiki` | Job context only | `LoggerAdapter.extra` |\n| `trace_id` | When OTel active | OTel span context |\n\n### Job-scoped logging\n\n```python\nfrom synthadoc.core.logging_config import get_job_logger\n\nlog = get_job_logger(__name__, job_id=\"abc123\", operation=\"ingest\", wiki=\"my-wiki\")\nlog.info(\"Page created: %s\", slug)\n# → {\"ts\": \"…\", \"level\": \"INFO\", \"logger\": \"…\", \"msg\": \"Page created: alan-turing\",\n# \"job_id\": \"abc123\", \"operation\": \"ingest\", \"wiki\": \"my-wiki\"}\n```\n\n### Setup (called once at server start)\n\n```python\nfrom synthadoc.core.logging_config import setup_logging\nsetup_logging(wiki_root=Path(\"/path/to/wiki\"), cfg=config.logs, verbose=False)\n```\n\nIdempotent — safe to call multiple times (subsequent calls are no-ops).\n\n### OpenTelemetry\n\nDefault: file exporter writing to `traces.jsonl`. Switch to any OTLP backend:\n\n```toml\n[observability]\nexporter = \"otlp\"\notlp_endpoint = \"http://localhost:4317\"\n```\n\nSpans cover: full operation tree (orchestrator → agent → LLM calls → storage writes), with token counts, cost, and cache hit/miss as span attributes.\n\n### Log level guidance\n\n| Level | When to use |\n|-------|------------|\n| `DEBUG` | LLM prompt bodies, cache key details, BM25 scores, entity extraction details |\n| `INFO` | Job lifecycle, page created/updated, server started, lint summary |\n| `WARNING` | Soft failures (network unreachable), suspicious patterns |\n| `ERROR` | Job failed, API error, file write failed |\n| `CRITICAL` | Server cannot start |\n\n---\n\n## 16. Security\n\n### Path traversal\n\n`WikiStorage` normalizes all paths with `Path.resolve()` and asserts each is a child of `wiki_root`. Raises `PermissionError` on violation.\n\n### Prompt injection\n\n- LLM output validated against a strict schema; unrecognized keys dropped silently\n- Slug blacklist: `wikilinks`, `wiki`, `obsidian`, `dataview`, `index`, `dashboard`, `log`, `audit`, `hooks`, `skills`\n- System prompt instructs the model to never follow instructions embedded in source documents\n\n### Network exposure\n\nHTTP and MCP servers bind to `127.0.0.1` at OS socket level. Not configurable. No remote access without a separate reverse proxy (which the user must explicitly set up).\n\n### HTTP DoS\n\n- Body limit: 10 MB (returns 413)\n- Concurrent request cap: 20 (asyncio semaphore)\n- Request timeout: 60 seconds\n\n### Audit trail\n\n`audit.db` is append-only in normal operation. The only deletion command is `jobs purge --older-than `, which only removes records older than the given threshold.\n\n### Custom skills trust model\n\nSkills in `/skills/` or `~/.synthadoc/skills/` run in the same Python process. This is intentional — the wiki root is a trusted location, analogous to `~/bin`. Do not point a wiki root at an untrusted directory.\n\n---\n\n## 17. Plugin Development Guide\n\nThis section is for developers building custom skills or LLM providers.\n\n### Writing a skill\n\n1. Create a skill folder in `/skills/` or `~/.synthadoc/skills/`.\n2. Add a `SKILL.md` with YAML frontmatter (name, version, entry, triggers, requires).\n3. Create `scripts/main.py` and subclass `BaseSkill` from `synthadoc.skills.base` (Apache-2.0 — no AGPL obligation).\n4. Implement `extract(source: str) -> ExtractedContent`.\n\n**Folder layout:**\n```\nslack_export/\n SKILL.md\n scripts/\n main.py\n references/\n format-notes.md ← optional; load with self.get_resource(\"format-notes.md\")\n```\n\n**`SKILL.md`:**\n```yaml\n---\nname: slack_export\nversion: \"1.0\"\ndescription: Extract messages from a Slack export ZIP\nentry:\n script: scripts/main.py\n class: SlackExportSkill\ntriggers:\n extensions: [\".slack.zip\"]\n intents: [\"slack export\", \"slack archive\"]\nrequires: []\n---\n\nLoads all JSON channel files from a Slack export ZIP and returns the message text.\n```\n\n**`scripts/main.py`:**\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.skills.base import BaseSkill, ExtractedContent\n\nclass SlackExportSkill(BaseSkill):\n\n async def extract(self, source: str) -> ExtractedContent:\n import zipfile, json\n messages = []\n with zipfile.ZipFile(source) as zf:\n for name in zf.namelist():\n if name.endswith(\".json\"):\n data = json.loads(zf.read(name))\n for msg in data:\n if \"text\" in msg:\n messages.append(msg[\"text\"])\n return ExtractedContent(\n text=\"\\n\".join(messages),\n source_path=source,\n metadata={},\n )\n```\n\n**Error handling:** Raise `ValueError` with a clear message if the source cannot be processed. Raise `ImportError` if an optional dependency is missing (the agent will surface a helpful message to the user).\n\n### Writing a provider\n\nBuilt-in providers: `anthropic`, `openai`, `gemini`, `groq`, `minimax`, `deepseek`, `qwen`, `ollama`. For any provider that exposes an OpenAI-compatible API, no custom class is needed — the built-in `openai` provider with a custom `base_url` is sufficient.\n\nFor a fully proprietary API, subclass `LLMProvider`:\n\n```python\n# SPDX-License-Identifier: MIT\nfrom synthadoc.providers.base import LLMProvider, Message, CompletionResponse\n\nclass MyProvider(LLMProvider):\n\n async def complete(\n self,\n messages: list[Message],\n model: str,\n temperature: float = 0.0,\n **kwargs,\n ) -> CompletionResponse:\n # Call your API …\n return CompletionResponse(\n text=\"…\",\n input_tokens=N,\n output_tokens=M,\n )\n```\n\nPlace in `~/.synthadoc/providers/` or the wiki `providers/` directory. Reference by name in config:\n\n```toml\n[agents]\ndefault = { provider = \"my_provider\", model = \"my-model-id\" }\n```\n\n### Writing a hook\n\nHooks can be in any language. They receive JSON on stdin and must exit 0 on success.\n\n```bash\n#!/usr/bin/env bash\n# hooks/notify.sh\ncontext=$(cat)\nevent=$(echo \"$context\" | jq -r '.event')\nwiki=$(echo \"$context\" | jq -r '.wiki')\necho \"Event $event fired on wiki $wiki\" | mail -s \"Synthadoc notification\" you@example.com\n```\n\n---\n\n## 18. Routing\n\n### ROUTING.md format\n\n`ROUTING.md` lives at `/ROUTING.md`. It groups page slugs under topic branch headings:\n\n```markdown\n## People\n- [[alan-turing]]\n- [[grace-hopper]]\n\n## Hardware\n- [[eniac]]\n- [[von-neumann-architecture]]\n```\n\n### RoutingIndex\n\n`RoutingIndex` parses the file, exposes a `branches: dict[str, list[str]]` mapping, and provides:\n\n- `parse(path)` — class method; returns an empty index if the file is absent\n- `validate(existing_slugs)` — returns `(branch, slug)` pairs present in ROUTING.md but not in the wiki\n- `clean(existing_slugs)` — removes dangling entries in-place; returns removed pairs\n- `add_slug(slug, branch)` — idempotent append\n- `slugs_for_branches(branch_names)` — flat list of slugs across the named branches\n- `save(path)` — serialises back to disk\n\n### Query routing\n\nWhen `routing_path` is passed to `QueryAgent`, each query first picks the 1–2 most relevant branches via a lightweight LLM call (returns `[]` on any failure) and restricts the BM25 corpus to those slugs. Falls back to full-corpus search when no branch is selected.\n\n### Ingest placement\n\nWhen `routing_path` is passed to `IngestAgent`, newly created pages are automatically placed into the most appropriate branch via a lightweight LLM call after the page is written.\n\n### Alias expansion\n\nPages may carry an `aliases:` list in YAML frontmatter. `QueryAgent._expand_aliases` replaces alias matches in the question with the canonical slug before BM25 search (longest alias first to avoid partial-match conflicts).\n\n### Protected scaffold zone\n\n`SCAFFOLD_MARKER = \"\"` separates user-authored content (above) from scaffold-managed content (below) in `wiki/index.md`. The `preserve_user_zone(existing, new_scaffold)` helper in `synthadoc.agents.scaffold_agent` preserves the user zone when re-running the scaffold command. If the marker is absent, scaffold rewrites the whole file (original behaviour).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc routing init` | Generate ROUTING.md from current index.md branch structure |\n| `synthadoc routing validate` | Report dangling slugs (dry run) |\n| `synthadoc routing clean` | Remove dangling slugs from ROUTING.md |\n\nAll commands accept `--wiki-root `.\n\n### Obsidian plugin\n\nThe `Routing: manage ROUTING.md...` command opens a `RoutingModal`. On open it calls `GET /routing/status` and enables or disables the three buttons accordingly:\n\n| State | Init | Validate | Clean |\n|---|---|---|---|\n| ROUTING.md absent | enabled | disabled | disabled |\n| ROUTING.md present | disabled | enabled | enabled |\n| Server unreachable | disabled | disabled | disabled |\n\nAfter each action the result appears in an inline result area with per-entry `[Branch] [[slug]]` detail rows. The ROUTING.md preview box (max-height 120 px, scrollable) is shown/refreshed by Init and Clean operations.\n\n---\n\n## 19. Candidates Staging\n\n### Staging policy\n\nNew pages can be routed to `wiki/candidates/` instead of `wiki/` based on the `[ingest] staging_policy` setting:\n\n| Value | Behaviour |\n|---|---|\n| `off` | All new pages go directly to `wiki/` (default) |\n| `all` | All new pages go to `wiki/candidates/` |\n| `threshold` | Pages below `staging_confidence_min` go to `wiki/candidates/` |\n\n`staging_confidence_min` values: `high` (default), `medium`, `low`. Confidence ordering: `high > medium > low`.\n\n### Exclusion from search\n\n`wiki/candidates/` is excluded from `WikiStorage.list_pages()` and therefore from BM25, lint, and contradiction detection. Candidates are invisible to all agents until promoted.\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc staging policy [off\\|all\\|threshold]` | Show or set the staging policy |\n| `synthadoc staging policy --min-confidence ` | Set minimum confidence threshold |\n| `synthadoc candidates list` | List all candidate pages with confidence and date |\n| `synthadoc candidates promote ` | Move a candidate to `wiki/` |\n| `synthadoc candidates promote --all` | Promote all candidates |\n| `synthadoc candidates discard ` | Delete a candidate |\n| `synthadoc candidates discard --all` | Delete all candidates |\n\nPolicy changes take effect on the next ingest job — no server restart needed.\n\n### HTTP API\n\n| Method | Path | Request | Response |\n|--------|------|---------|----------|\n| `GET` | `/staging/policy` | — | `{policy: str, confidence_min: str\\|null}` |\n| `POST` | `/staging/policy` | `{policy: str, confidence_min?: str}` | `{policy: str, confidence_min: str\\|null}` |\n| `GET` | `/candidates` | — | `[{slug: str, title: str, confidence: str, ingested_at: str}]` |\n| `POST` | `/candidates/promote-all` | — | `{promoted: int, updated: int}` |\n| `POST` | `/candidates/discard-all` | — | `{discarded: int}` |\n| `POST` | `/candidates/{slug}/promote` | — | `{promoted: slug, new: bool, updated: bool}` |\n| `POST` | `/candidates/{slug}/discard` | — | `{discarded: slug}` |\n\nPromote moves the file from `wiki/candidates/.md` to `wiki/.md`. If a page with the same slug already exists in `wiki/` (a staged update to an existing page), the existing file is overwritten. Only newly created pages (not overwrites) are indexed into BM25.\n\n### Obsidian plugin\n\nThe **Staging: manage staging policy...** command opens `StagingModal`:\n\n- A status block shows the current policy in plain language (e.g. \"Staging is **enabled (threshold)**. Pages below *high* confidence are staged.\")\n- A segmented control switches between **Off** / **All** / **Threshold**.\n- When **Threshold** is selected, a second segmented control sets **Min confidence**: **High** / **Medium** / **Low**.\n- **Save** calls `POST /staging/policy` and refreshes the status block inline.\n- A footer link **Candidate pages →** closes the modal and opens `CandidatesModal`.\n\nThe **Candidates: review candidate pages...** command opens `CandidatesModal`:\n\n- Loads all candidates via `GET /candidates` and displays them in a paginated table (50 rows per page).\n- Each row has a checkbox, the page slug, a colour-coded confidence badge (green = high, amber = medium, red = low), and the ingest timestamp.\n- A select-all checkbox in the header checks or clears every row on the current page.\n- **Promote All** / **Discard All** operate on every candidate regardless of page; **Promote Selected** / **Discard Selected** operate on checked rows only. The table reloads automatically after each action.\n- A footer link **← Staging policy** closes the modal and opens `StagingModal`.\n\n---\n\n## 20. Context Packs\n\n### ContextAgent\n\n`ContextAgent` builds a token-bounded evidence pack from the wiki:\n\n1. Decomposes the goal into sub-questions (reuses `QueryAgent.decompose`)\n2. Runs BM25 hybrid search per sub-question in parallel\n3. Merges results, keeping the best score per slug\n4. Packs pages greedily within the token budget (word count / 0.75 approximation)\n5. Records omissions when the budget is exhausted\n\nConstructor: `ContextAgent(provider, store, search, token_budget=4000, top_n=8)`\n\nMethod: `await agent.build(goal, token_budget=None) → ContextPack`\n\n### ContextPack\n\n| Field | Type | Description |\n|---|---|---|\n| `goal` | `str` | The input goal string |\n| `token_budget` | `int` | Effective budget used |\n| `tokens_used` | `int` | Tokens consumed by included pages |\n| `pages` | `list[ContextPage]` | Included pages, ranked by relevance |\n| `omitted` | `list[ContextPage]` | Pages excluded due to budget |\n\n`ContextPack.to_markdown()` renders a human-readable evidence pack. `ContextPack.to_dict()` returns a JSON-serialisable dict for the REST API.\n\n### Default token budget\n\nThe default token budget is configured via `[query] context_token_budget` in `config.toml` (default: 4000). The HTTP request body and CLI `--tokens` flag can override it per call.\n\n### REST API\n\n```\nPOST /context/build\nContent-Type: application/json\n\n{\"goal\": \"early computing pioneers\", \"token_budget\": 2000}\n```\n\nResponse: `ContextPack.to_dict()` — keys `goal`, `token_budget`, `tokens_used`, `pages`, `omitted`.\n\n### CLI command\n\n```bash\n# Print to terminal — inspect, copy, or pipe into another tool\nsynthadoc context build \"early computing pioneers\"\n\n# Custom token budget (default 4000)\nsynthadoc context build \"early computing pioneers\" --tokens 2000\n\n# Save to a file — feed to an external LLM prompt or store next to a document you're writing\nsynthadoc context build \"early computing pioneers\" --output briefing.md\n```\n\n---\n\n## 21. Adversarial Review\n\n### Concept\n\nStandard lint validates wiki structure — contradictions, orphans, broken links. It does not evaluate whether the *content* of a page is accurate. The adversarial review closes this gap: after structural checks complete, a second independent LLM pass interrogates every page for epistemic overreach — overstated claims, unsupported assertions, and high-confidence statements the source material does not support.\n\nThe key architectural decision is cross-model independence. When the adversarial reviewer is a different model family from the ingest model, neither shares the training-induced inductive biases that cause same-model self-review to systematically miss the same class of errors.\n\n### LintAgent integration\n\nThe adversarial review runs as the final phase of every `synthadoc lint run`. After orphan detection and contradiction checks complete, `LintAgent` calls `_adversarial_single(slug, content)` for every non-excluded page concurrently via `asyncio.gather()`. A 100-page wiki completes in the same wall-clock time as a single LLM call.\n\nEach `_adversarial_single` call prompts the adversarial model to act as a skeptical editor and return a JSON array of `{claim, concern}` objects. Results are capped at `adversarial_max_per_page` (default 2) per page. Failures are caught per-page — rate-limit errors and parse failures are stored as non-fatal warning entries and never abort the lint job.\n\nWhen `--no-adversarial` is passed to `lint run`, the adversarial phase is skipped entirely and any existing `lint_warnings` are cleared from all page frontmatter.\n\nWhen `--no-lifecycle` is passed to `lint run`, all four lifecycle checks are skipped. Existing `page_states` and `lifecycle_events` records are not modified.\n\n### `lint_warnings` frontmatter\n\nWarnings are written directly to each page's YAML frontmatter after each lint run:\n\n```yaml\nlint_warnings:\n - claim: \"Saved over fourteen million lives.\"\n concern: \"This figure lacks scholarly consensus — historians dispute both the\n precision and the causal attribution to Turing's cryptanalysis alone.\"\n - claim: \"Most consequential business decision of the era.\"\n concern: \"An unsupported superlative — the MS-DOS licence retention and Intel's\n exclusive CPU supply deal were equally pivotal.\"\n```\n\nThe field is absent when no warnings exist. Cleared automatically when `--no-adversarial` is used, ensuring stale warnings do not persist after the pass is disabled.\n\n### Configuration\n\n```toml\n# config.toml\n[agents]\nlint = { provider = \"minimax\", model = \"MiniMax-M2.5\" }\nadversarial = { provider = \"anthropic\", model = \"claude-sonnet-4-6\" } # independent judge — different model family\n\n[lint]\nadversarial_max_per_page = 2 # raise to 3–5 for a deeper audit; lower to 1 for less noise\n```\n\n`[agents].adversarial` falls back to `[agents].default` if absent — the adversarial pass always runs, it just uses the same model as ingest (less effective, still useful).\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc lint run` | Full lint pass including adversarial review |\n| `synthadoc lint run --no-adversarial` | Structural-only lint; clears existing `lint_warnings` |\n| `synthadoc lint report` | Show warnings — CLI output has a dedicated Adversarial section |\n\n### HTTP API\n\n`GET /lint/report` returns a `LintReport` object. The `adversarial_warnings` field carries all warnings across all pages:\n\n```json\n{\n \"adversarial_warnings\": [\n {\n \"slug\": \"alan-turing\",\n \"claim\": \"Saved over fourteen million lives.\",\n \"concern\": \"This figure lacks scholarly consensus…\"\n }\n ]\n}\n```\n\nEmpty list when no warnings exist or the pass was skipped.\n\n### Obsidian plugin\n\n`Synthadoc: Lint: run...` modal adds a **Skip adversarial review** checkbox alongside the existing **Auto-resolve** checkbox. When ticked, the lint job runs structural checks only and clears stale warnings.\n\n`Synthadoc: Lint: report` is a 3-tab modal — **Contradictions**, **Orphans**, **Adversarial**. The Adversarial tab renders each warning with the flagged claim in orange, the concern below it in muted text, and suggested re-ingest commands derived from the page's source files.\n\n---\n\n## 22. Claim-Level Provenance\n\n### Concept\n\nEvery compiled wiki page is a synthesis — the LLM reads source documents and rewrites them as prose. Claim-level provenance closes the audit gap: during ingest, a dedicated annotation pass inserts a `^[filename:L-L]` citation marker at the end of each substantive paragraph, mapping the compiled claim to the exact line range in the raw source that supports it. Markers are stored in the page body, validated by lint, and recorded in `audit.db`. In Obsidian they render as interactive chips — one click opens the Source Viewer.\n\n### IngestAgent Pass 4 — `_annotate_citations()`\n\nCalled within the Write pass for each page section immediately before it is appended to the page. The LLM receives:\n\n1. The numbered raw source text (lines 1, 2, 3 … N)\n2. The compiled section to annotate\n\nIt returns the section with `^[filename:L-L]` markers appended to substantive paragraphs. The result is validated against a sanity check (markers must reference real line numbers in the source). On any failure — LLM error, parse failure, or sanity check — the original un-annotated section is used and the failure is recorded as a `citation_pass4_skipped` audit event. Ingest always completes.\n\nResults are cached by section SHA-256 so re-ingest of unchanged sections does not incur an extra LLM call.\n\n### Sidecar files\n\nTo support the Source Viewer in Obsidian, `_write_sidecar()` writes two files to `.synthadoc/extracted/` for every locally ingested source:\n\n| File | Contents | Source types |\n|------|----------|-------------|\n| `.txt` | Plain UTF-8 extracted text with line numbers preserved | All local file types |\n| `.pagemap.json` | JSON array mapping line numbers to PDF page numbers | PDF only |\n\nThe pagemap enables the \"Open PDF at page N →\" button in the Source Viewer to resolve a line range to the correct PDF page without re-parsing the document. Web and YouTube sources do not produce sidecars (no stable local path to key on).\n\n### `claim_citations` table\n\nStored in `audit.db`. Written by `AuditDB.record_claim_citations()` after each annotated page section is saved.\n\n| Column | Type | Notes |\n|--------|------|-------|\n| `id` | INTEGER PK | |\n| `page_slug` | TEXT | Wiki page the citation belongs to |\n| `source_file` | TEXT | Basename of the raw source file |\n| `line_start` | INTEGER | First line of the supporting passage |\n| `line_end` | INTEGER | Last line of the supporting passage |\n| `claim_excerpt` | TEXT | First ~100 chars of the annotated paragraph |\n| `ingested_at` | TEXT | UTC ISO-8601 |\n\n### HTTP API\n\n```\nGET /provenance/citations\n ?page= filter by wiki page\n &source= filter by source file\n &broken= return only citations that failed validation\n &limit=N page size (default 50)\n &offset=N pagination offset\n &sort= ingested_at | page_slug | source_file (default: ingested_at)\n &order=asc|desc (default: desc)\n```\n\nResponse: `{total: int, citations: [CitationRow]}`\n\n### CLI commands\n\n| Command | Description |\n|---|---|\n| `synthadoc audit citations -w ` | Last 50 citations across the whole wiki |\n| `synthadoc audit citations -w --page ` | All citations for one page |\n| `synthadoc audit citations -w --broken` | Citations that failed line-range validation |\n\n### Obsidian plugin\n\n**Citation chips (Reading View only):** The Obsidian post-processor transforms `^[filename:L-L]` inline footnote markers into styled chips rendered after the claim. Chips only appear in Reading View (`Ctrl/Cmd+E`) — not in Edit or Live Preview mode.\n\n**Source Viewer modal:** Clicking a chip opens a draggable modal showing the referenced source lines highlighted with ±5 lines of surrounding context. File resolution order:\n\n1. `.synthadoc/extracted/.txt` — pre-extracted sidecar (all local types)\n2. `raw_sources/` — direct fallback for plain-text types (`.md`, `.txt`, `.csv`)\n3. Friendly error for binary types (`.xlsx`, etc.) with instructions to open the original\n\nFor PDF sources, if the pagemap sidecar exists and the target page is > 1, a **\"Open PDF at page N →\"** button closes the Source Viewer and opens the PDF at the correct page in Obsidian's native viewer.\n\n**Page Provenance modal (`Synthadoc: View Page Provenance`):** A sortable, paginated table of every citation across the wiki. Columns: Page, Claim, Source, Lines, Ingested. Sort by any column header; filter by slug or source filename. Pagination is pinned below the table and always visible. Click any row to open the Source Viewer for that citation. All cell content can be selected and copied independently of the row-click action.\n\n---\n\n\n## 23. Lifecycle Machine\n\n### Concept\n\nEvery wiki page moves through a defined set of states that reflect its review status and the health of its source material. Pages start as `draft` — compiled but not yet validated — and advance to `active` when lint passes all checks. Subsequent changes to the source file on disk push the page to `stale`; a missing source file triggers `archived`. Manual transitions allow operators to override any state.\n\n### States\n\n| State | Meaning | How to reach it |\n|---|---|---|\n| `draft` | Newly compiled, not yet lint-reviewed | Automatic on ingest |\n| `active` | Lint-reviewed, current, trusted | Lint auto-promotes from `draft` |\n| `contradicted` | Conflict detected | Lint detects contradiction between sources |\n| `stale` | Source file changed since last ingest | Lint detects SHA-256 hash mismatch |\n| `archived` | Source removed or explicitly retired | Lint auto-archives on missing source; or manual |\n\n### Transition rules\n\n| From | To | Trigger | Condition |\n|---|---|---|---|\n| _(none)_ | `draft` | `ingest` | New page created by ingest |\n| `draft` | `active` | `lint` | Page passes all lint checks |\n| `draft` / `stale` / `contradicted` | `active` | `cli` / `api` | Manual activate (`synthadoc lifecycle activate`) |\n| `active` | `contradicted` | `lint` | Contradiction detected between two or more sources |\n| `active` | `stale` | `lint` | Local source: SHA-256 hash mismatch; or URL source older than `url_staleness_days` |\n| `active` / `stale` | `archived` | `lint` | Local source no longer exists on disk; or URL source returns 404/410 (opt-in) |\n| any | `archived` | `cli` / `api` | Manual archive (`synthadoc lifecycle archive`) |\n| `archived` | `draft` | `cli` / `api` | Manual restore (`synthadoc lifecycle restore`) — places page back in review queue |\n| `contradicted` | `archived` | `cli` / `api` | Manual archive after reviewing the conflict |\n\n### Storage\n\nTwo new tables in `audit.db`:\n\n**`page_states`** — fast slug-keyed current state index (one row per page):\n\n```sql\npage_states (slug TEXT PK, state TEXT, updated_at TEXT, triggered_by TEXT)\n```\n\n**`lifecycle_events`** — immutable append-only audit log:\n\n```sql\nlifecycle_events (id INTEGER PK, slug TEXT, from_state TEXT, to_state TEXT,\n reason TEXT, triggered_by TEXT, timestamp TEXT)\n```\n\n`triggered_by` values: `ingest`, `lint`, `cli`, `api`.\n\n### LintAgent integration\n\nFour lifecycle checks run at the end of every lint pass, after all existing checks, unless `--no-lifecycle` is passed:\n\n1. **Archived detection** — source no longer available → transition page to `archived`\n2. **Stale detection** — source has changed since last ingest → transition page to `stale`\n3. **Draft promotion** — `draft` page with no active issues → transition to `active`\n4. **Manual-edit sync** — frontmatter `status` ≠ `page_states` DB → reconcile DB record to match\n\nPass `--no-lifecycle` to `synthadoc lint run` to skip all four checks. Existing `page_states` and `lifecycle_events` records are not modified.\n\n#### Check 1 — Archived detection (local and URL sources)\n\nFor **local file sources**: if the source path recorded in frontmatter no longer exists on disk, the page is transitioned to `archived`.\n\nFor **URL sources** (`http://`, `https://`, `youtube.com/watch?v=…`): availability is checked only when `[lint] check_url_availability = true` (default: `false` — opt-in because it adds a network call per URL source during every lint run).\n\n- **Generic URLs** — an HTTP HEAD request is issued. Responses of 404 or 410 are treated as archived. Timeouts, connection errors, and any other status code leave the page unchanged (conservative: no false positives on transient failures).\n- **YouTube URLs** — the transcript API is probed with the video ID. A `VideoUnavailable` response means the video is deleted or private → `archived`. Any other error (network error, parsing failure) leaves the page unchanged.\n\nEnable with the `--check-urls` flag or the config key:\n\n```bash\nsynthadoc lint run --check-urls\n```\n\n```toml\n[lint]\ncheck_url_availability = true # default: false\n```\n\n#### Check 2 — Stale detection (local and URL sources)\n\nFor **local file sources**: a SHA-256 hash of the current file on disk is compared to the hash recorded at ingest time. A mismatch transitions the page to `stale`.\n\nFor **URL sources**: staleness is age-based. If `url_staleness_days` is non-zero, the `ingested_at` timestamp from `audit.db` is compared to the current time. Pages whose last ingest is older than the threshold are transitioned to `stale`, prompting a re-ingest.\n\n```toml\n[audit]\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n```\n\nURL staleness detection runs on every lint pass when the config value is non-zero — no extra flag required.\n\n#### Debug logging\n\nWhen URL availability or staleness checks run, the lint agent emits `DEBUG`-level log lines for each check outcome:\n\n```\nlifecycle url-check [youtube] id=dQw4w9WgXcQ url=https://www.youtube.com/watch?v=dQw4w9WgXcQ → unavailable\nlifecycle url-check [head] url=https://example.com/page → status=404 → unavailable\nlifecycle url-stale url=https://example.com/page → age=102d threshold=90d → stale\n```\n\nEnable debug logging in `config.toml`:\n\n```toml\n[logs]\nlevel = \"DEBUG\"\n```\n\n### Auto-retention\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\n```\n\nWhen non-zero, events older than `lifecycle_retention_days` are pruned from `lifecycle_events` at the end of each lint run. `page_states` records are never pruned — they represent current state, not history.\n\n### CLI commands\n\n```\nsynthadoc status -w \n Show page counts by lifecycle state alongside existing page and job totals.\n\nsynthadoc lint run [--no-lifecycle] [--check-urls]\n Run lint. --no-lifecycle skips all four lifecycle checks.\n --check-urls enables HTTP availability checks for URL sources (overrides config).\n\nsynthadoc lifecycle activate -w [--reason \"...\"]\n Transition a page to active.\n\nsynthadoc lifecycle archive -w [--reason \"...\"]\n Transition a page to archived.\n\nsynthadoc lifecycle restore -w [--reason \"...\"]\n Transition an archived page back to draft.\n\nsynthadoc lifecycle log [slug] -w [--state ]\n Print the event log for one page (or all pages). Filter by to_state with --state.\n\nsynthadoc audit lifecycle purge -w --before \n Delete lifecycle events older than (ISO-8601, e.g. 2026-01-01).\n\nsynthadoc audit lifecycle purge -w --keep-latest \n Keep only the most recent events per slug, delete the rest.\n```\n\n### HTTP API\n\n| Method | Path | Description |\n|--------|------|-------------|\n| `GET` | `/lifecycle/status` | Current state counts from `page_states` |\n| `GET` | `/lifecycle/events` | Paginated event log (`slug`, `to_state`, `limit`, `offset` query params) |\n| `POST` | `/lifecycle/transition` | Body: `{slug, to_state, reason?}` — validates allowed transition, writes both tables |\n\n### Obsidian plugin\n\n**`Synthadoc: Manage Page Lifecycle`** — opens `LifecycleModal`:\n\n- Loads all pages via `GET /lifecycle/status` and lists them in a sortable, paginated table (25 per page by default).\n- State filter checkboxes (one per state) narrow the table to the selected states.\n- Column headers (Slug, State, Last transition) are sortable — click to cycle ascending/descending/unsorted.\n- Each row shows valid action buttons for the current state. Click a button to trigger a transition — `ReasonModal` appears first, prompting for an optional reason string, before committing via `POST /lifecycle/transition`.\n- Draft and stale badge links on the lint modal and jobs panel open `LifecycleModal` pre-filtered to that state.\n\n### Configuration\n\n```toml\n[audit]\nlifecycle_retention_days = 365 # 0 = keep forever (default)\nurl_staleness_days = 90 # 0 = never mark URL sources stale (default)\n\n[lint]\ncheck_url_availability = true # default: false — adds a network call per URL source during lint\n```\n\n---\n\n## 24. Export Formats\n\nThe `synthadoc export` command serializes the wiki in four machine-readable formats, assembled server-side from cached data with zero additional LLM calls. Requires `synthadoc serve` to be running.\n\n### Formats\n\n**`llms.txt`** — Navigation index per the [llmstxt.org](https://llmstxt.org/) spec. Active pages appear under `## Pages` with a one-line description; contradicted and stale pages appear under `## Needs Review` with a reason note; archived pages are omitted entirely. With `--status active` only `## Pages` is emitted.\n\n**`llms-full.txt`** — Flat content dump. Pages are separated by `---`. Each page opens with `Status: | Confidence: | Tags: ...`. Provenance footnotes (`^[source.txt:42-58]`) are preserved verbatim in the body. No size limit — the full wiki is always exported. For very large wikis a streaming export path is planned as a future enhancement.\n\n**`graphml`** — Standard GraphML 1.1. Nodes = pages; edges = wikilinks extracted from page bodies. Node attributes:\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `label` | string | Human-readable page title — read by Gephi and Cytoscape as the display label |\n| `title` | string | Same as `label`; retained for backwards compatibility |\n| `status` | string | Lifecycle state (`draft`, `active`, `contradicted`, `stale`, `archived`) |\n| `confidence` | string | Ingest confidence level |\n| `orphan` | boolean | `true` if the page has zero inbound wikilinks |\n| `citation_count` | int | Reserved; always `0` in current release |\n| `inbound_link_count` | int | Number of other pages that link to this page |\n| `routing_branch` | string | Branch name from ROUTING.md membership |\n\nAll edges carry `edge_type=\"wikilink\"`. Self-links are suppressed. The file also embeds a `y:ShapeNode/y:NodeLabel` element (yEd namespace) so node labels render natively when the file is opened in **yEd Graph Editor**. No position data is embedded — run the tool's layout algorithm after import. Tested tools: yEd (Layout → Organic or Hierarchical), Gephi (enable labels via the Aα button in the bottom toolbar; run ForceAtlas2), Cytoscape (File → Import → Network from File).\n\n**`json`** — Agent-ready structured dump. Each page object contains:\n\n| Field | Description |\n|-------|-------------|\n| `claims[]` | Source file, line range, claim excerpt — from the claim provenance audit database |\n| `lifecycle_history[]` | Every state transition with `from`, `to`, `timestamp`, `triggered_by`, `reason` |\n| `ingest_cost_usd` | Cumulative LLM cost (USD) across all source files that contributed to this page |\n| `ingest_tokens` | Cumulative token count across all ingest calls for this page |\n| `sources[]` | Source file metadata: file, hash, size, ingested timestamp |\n| `lint_warnings[]` | Adversarial review findings: `{claim, concern}` pairs |\n\nWiki-level fields: `total_compilation_cost_usd`, `routing.branch_memberships`, `exported_at`, `page_count`.\n\n### CLI\n\n```bash\nsynthadoc export --format [--output ] [--status ] [--context-pack ] [--wiki ]\n```\n\nOutputs to stdout by default; `--output` writes to a file. `--status` filters pages by lifecycle state (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`).\n\n### POST /export endpoint\n\nAccepts `{ format, status_filter }`. Returns raw content with appropriate `Content-Type` (`text/plain; charset=utf-8` for text formats, `application/xml` for graphml, `application/json` for json). Returns 422 for unknown format. No LLM calls.\n\n### Obsidian\n\n**`Synthadoc: Export Wiki`** opens a modal with:\n- A brief description panel explaining each format\n- Format dropdown (json / llms.txt / llms-full.txt / graphml)\n- Output path field (full-width, pre-filled with today's date and correct extension, editable)\n- Status filter dropdown (`all` / `active` / `draft` / `stale` / `contradicted` / `archived`)\n- **Export** button — writes to the vault's `exports/` folder and opens the file automatically\n- **View Graph** button (graphml only) — opens an inline Cytoscape.js graph preview before saving\n\n---\n\n## 25. Streaming Query and Query Cache\n\n### SSE Streaming Query\n\n`GET /query/stream?q=[&session_id=][&no_cache=true]` returns a Server-Sent Events stream. Each event is a JSON-encoded object:\n\n```\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"retrieving\"}}\n\ndata: {\"event\": \"status\", \"data\": {\"phase\": \"synthesizing\", \"sources\": 3}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \"Alan\"}}\n\ndata: {\"event\": \"token\", \"data\": {\"text\": \" Turing\"}}\n\ndata: {\"event\": \"citations\", \"data\": {\"citations\": [\"alan-turing\", \"enigma\"]}}\n\ndata: {\"event\": \"done\", \"data\": {\"next_hints\": [\"What came after Turing?\", ...]}}\n```\n\n| Event | Payload | When |\n|---|---|---|\n| `status` | `{\"phase\": \"retrieving\"}` | Phase 1 starts |\n| `status` | `{\"phase\": \"synthesizing\", \"sources\": N}` | Phase 2 starts |\n| `token` | `{\"text\": \"…\"}` | Each LLM token |\n| `citations` | `{\"citations\": […]}` | After last token |\n| `gap` | `{\"suggested_searches\": […]}` | If knowledge gap detected |\n| `clarify` | `{\"prompt\": \"…\", \"candidates\": [\"slug-1\", …], \"action\": \"…\"}` | Action agent needs disambiguation (e.g. which page to activate) |\n| `notice` | `{\"text\": \"…\"}` | System message (e.g. conversation history was compressed) |\n| `done` | `{\"next_hints\": […]}` | Stream complete |\n| `error` | `{\"message\": \"…\"}` | On any exception |\n\nThe CLI `synthadoc query` renders tokens as they arrive using ANSI cursor control. Pass `--no-stream` to fall back to the blocking `POST /query` endpoint and print the full answer when complete.\n\nThe streaming path shares the same query decomposition, BM25 retrieval, and knowledge-gap detection as the blocking path. The only difference is delivery mechanism — SSE for streaming, plain JSON for blocking.\n\n### Epoch-Based Query Result Cache\n\nQuery answers are cached in `cache.db` under a composite key:\n\n```\ncache_key = hash(\n normalised_question,\n wiki_epoch,\n cache_version\n)\n```\n\n**Wiki epoch** is an integer stored in `cache.db` that increments on every event that changes wiki content:\n\n| Event | Effect on epoch |\n|-------|----------------|\n| `ingest` completes (pages written) | epoch + 1 |\n| `lifecycle transition` (any state change) | epoch + 1 |\n| `cache clear` | epoch reset to 0 |\n\nBecause the epoch is part of the cache key, any structural change to the wiki automatically invalidates all cached query answers — there is no explicit expiry TTL. Answers cached before an ingest are never served after it.\n\n**`--no-cache` flag:** bypasses `cache.get()` and `cache.set()` for the current call. The existing cache entry (if any) is left intact; it will be served on subsequent calls without `--no-cache`.\n\n**`synthadoc cache clear`:** deletes all entries from `cache.db`, including both ingest response cache and query answer cache. The epoch is reset to 0.\n\n### QueryAgent integration\n\n`QueryAgent.query()` checks the cache before decomposing the question. On a hit, the cached `QueryResult` is returned immediately — no BM25 search, no LLM call. On a miss, the full pipeline runs and the result is written to cache before returning.\n\nThe streaming endpoint inverts this: if a cache hit exists, the cached answer is replayed as an SSE stream (one token per event, then `[DONE]`), giving the same streaming UX even for cached responses.\n\n---\n\n## 26. Web Chat UI and Session Management\n\n### Architecture\n\nThe web chat UI is a React single-page application served at `GET /app` by the Synthadoc HTTP server. It communicates with the existing REST API — no additional server process or port is required.\n\n**Serving:** The bundled static assets (`index.html`, `main.js`, `main.css`) are packaged with the `synthadoc` Python distribution under `synthadoc/web_ui/dist/`. The HTTP server mounts them at `/app` via a static file handler. The SPA's API calls target the same origin (`http://127.0.0.1:`), so no CORS configuration is needed beyond what is already in place for the Obsidian plugin.\n\n### Session Management\n\nEach browser session is assigned a `session_id` (UUID) on `POST /sessions`. The server maintains a lightweight in-memory store per session:\n\n| Field | Type | Description |\n|---|---|---|\n| `session_id` | UUID | Unique identifier for this browser session |\n| `mode` | str | Session mode (`NEW_WIKI`, `EXPLORER`, `HEALTH_CHECK`, `POWER_USER`) |\n| `cursor` | int | Current position in the hint pool for windowed rotation |\n| `last_hints` | list[str] | Hints returned in the previous response (used for deduplication) |\n\n**Session mode** is derived from the wiki's current state at `POST /sessions`:\n\n| Mode | Condition | Hint behaviour |\n|---|---|---|\n| `NEW_WIKI` | `WikiStorage.count_pages() < 5` | Onboarding chips — guides user through first ingest |\n| `EXPLORER` | ≥5 pages, no prior `chat_sessions` rows | Discovery chips — broad overview questions |\n| `HEALTH_CHECK` | ≥5 pages, prior sessions, ≥1 `stale` page | Lifecycle chips — suggests running lint or reviewing stale pages |\n| `POWER_USER` | ≥5 pages, prior sessions, no stale pages | Context-sensitive follow-up chips |\n\nThe mode is returned by `POST /sessions` and also visible as a badge in the UI header.\n\n### Hint Engine\n\nThe `HintEngine` class (server-side, no LLM call) generates contextual chips:\n\n**Initial hints** (`initial_hints(mode)` — shown before the first question):\n3 fixed chips selected from the mode's priority hint set (e.g. `NEW_WIKI` → \"How do I ingest my first document?\").\n\n**After-response hints** (`after_response_windowed(answer, mode, cursor, previous_hints)`):\nUses a sliding window over a deduplicated hint pool to rotate suggestions across sessions. The pool is built by placing the current mode's hints first, followed by other modes' hints (no duplicates). A `cursor` advances by 3 on each call so consecutive responses cycle through the full pool over time.\n\nIf the answer text matches a topic pattern (e.g. the word \"stale\" in the answer triggers \"How do I run a lint check?\"), topic-relevant hints override the window — unless they exactly match the previous response's hints, in which case the pool window is used instead. This prevents keyword-triggered hints from repeating on consecutive wiki-management answers.\n\nHint chips are rendered below each answer and in the empty-state panel before the first question. Clicking a chip populates the text box without auto-submitting.\n\n### Multi-turn Conversation\n\nEach `GET /query/stream` call may include a `session_id` (UUID from `POST /sessions`). When a session_id is present, the HTTP server loads conversation history from `audit.db` (up to `conversation_history_turns` most recent turns, default 10) and passes it to `QueryAgent.run_stream()`.\n\n**Follow-up question rewriting.** When history is non-empty, a `RewriteAgent` rewrites the user's follow-up question into a self-contained form before BM25 retrieval. This converts context-dependent phrases (\"What came after that?\" or \"Tell me more about his early life\") into standalone questions (\"What came after Alan Turing's work at Bletchley Park?\") so keyword retrieval targets the right pages.\n\n**History overflow compression.** When the accumulated session exceeds `conversation_history_turns`, a `SummarizeAgent` compresses the oldest turns into a single `[Session summary: …]` assistant turn. A `notice` SSE event is emitted the first time compression occurs in a session, so the user can see that earlier context was condensed.\n\n**Clarify events.** When the action agent detects an action-intent query (e.g. \"Activate a draft page\") but cannot resolve which specific page the user means, it sets `needs_clarification=True` and returns a prompt and candidate list. The HTTP server emits a `clarify` SSE event instead of routing to the synthesis pipeline. The web UI renders this as numbered chip buttons (one per candidate page) plus a free-text hint, letting the user tap a chip or type a page name directly.\n\n**History persistence.** Conversation turns are written to `audit.db` per session. This means history survives a server restart — the same session_id can resume where it left off. Sessions older than `session_max_age_hours` (default 24 h) are purged by the hourly background cleanup task.\n\n### Session history sidebar\n\nThe left navigation bar in the web UI is driven by `GET /sessions` (returns up to 20 recent sessions, ordered by `last_active DESC`) rather than `localStorage`, so history is consistent across browser tabs and survives page refreshes.\n\n**2-level collapsible tree:**\n- Sessions with a single user turn appear as a flat entry showing the question text and relative timestamp.\n- Sessions with two or more user turns render as a collapsible group: the root row shows the first question plus a turn count badge (e.g. `3 turns`); a **▸** chevron toggles expansion; expanded child rows show each follow-up question with a `↳` indent.\n\n**Session restore:** clicking any session root or child turn calls `GET /sessions/{session_id}/messages` to hydrate the full message list, then sets the active `session_id` in the query stream hook so subsequent questions continue the same conversation thread.\n\n**New Run:** the **+ New Run** button calls `POST /sessions` to allocate a fresh `session_id` and resets the chat window to the empty-state hero screen.\n\n**API surface:**\n\n| Endpoint | Description |\n|---|---|\n| `GET /sessions?limit=N` | Returns `[{session_id, mode, created_at, last_active, turns: [str]}]` — `turns` is the list of user message contents in chronological order |\n| `GET /sessions/{session_id}/messages` | Returns `[{role, content}]` for every message in the session, oldest first |\n\n**Configuration:**\n\n```toml\n[query]\nconversation_history_turns = 10 # turns to include in each request (default: 10; 0 = disable history)\n```\n\n### CLI command\n\n```\nsynthadoc web [-w wiki] [--port N]\n```\n\nOpens the default browser to `http://localhost:{port}/app`. The server must already be running (`synthadoc serve`). This command is a thin wrapper around the OS `open`/`start`/`xdg-open` call — it does not start a new server process.\n\n---\n\n## Appendix A — Release Feature Index\n\n### v0.1.0 (Community Edition)\n\n- **3 agents** — IngestAgent (two-step cached synthesis), QueryAgent (BM25 + LLM), LintAgent (contradiction + orphan detection + auto-resolution)\n- **9 built-in skills** — PDF, URL, Markdown/TXT, DOCX, PPTX, XLSX/CSV, Image (vision), Web search (Tavily), YouTube transcript\n- **Folder-based skill system** — each skill is a self-contained folder with a `SKILL.md` manifest; intent-based dispatch alongside extension matching; drop a folder in `skills/` to add a new format without touching core code\n- **2 access surfaces** — CLI (thin HTTP client), HTTP REST API\n- **Obsidian plugin** — ingest (file picker, URL, all sources, web search), query modal, lint report, jobs list — all from the command palette; ribbon shows engine health + page count\n- **8 LLM providers** — Anthropic, OpenAI, Gemini (free tier), Groq (free tier), MiniMax (paid, multimodal), DeepSeek (paid, very cheap text-only), Qwen (paid DashScope cloud), Ollama (local); switch with one config line\n- **Two-step ingest** — `_analyse()` caches entity extraction + summary; decision prompt uses summary instead of full text; reduces cost on large documents\n- **purpose.md scope filtering** — define what belongs in your wiki; the LLM skips out-of-scope sources cleanly\n- **overview.md auto-summary** — 2-paragraph wiki overview regenerated automatically after every ingest\n- **Audit CLI** — `synthadoc audit history / cost / events` query `audit.db`; `--analyse-only` flag previews ingest analysis before writing pages\n- **3-layer cache** — embedding cache, LLM response cache, provider prompt cache\n- **Cost guards** — configurable soft-warn and hard-gate USD thresholds\n- **Hook system** — shell commands on `on_ingest_complete` and `on_lint_complete` lifecycle events; blocking or background; context passed as JSON on stdin\n- **Job queue** — SQLite-backed, persistent, retry with exponential backoff; `failed` vs `dead` status distinction\n- **Multi-wiki** — unlimited isolated wikis, each on its own port\n- **OpenTelemetry** — traces, metrics, structured logs; OTLP export optional\n- **Cross-platform** — Windows, Linux, macOS\n\n### v0.2.0 (Community Edition)\n\n- **Query decomposition** — `QueryAgent.decompose()` breaks complex questions into 1–N focused sub-questions (cap=4); parallel BM25 search per sub-question; merged and deduplicated by highest score; graceful fallback on LLM error; markdown fence stripping for cross-model robustness\n- **Query audit trail** — `queries` table in `audit.db`; every query recorded with question text, sub-question count, tokens, cost, timestamp; `cost_summary()` now aggregates ingest + query spend; exposed via `GET /audit/queries`, `synthadoc audit queries`, and the Obsidian `Audit...` modal (Query history tab)\n- **Per-model cost tracking** — per-token rate table covers all 5 providers; cost calculated for both ingest and query operations and stored in `audit.db`; Ollama records no API cost (local model); unknown models use a conservative fallback rate; exposed via `audit cost` CLI and `GET /audit/costs`\n- **Knowledge gap detection** — three independent signals (too few pages, low BM25 max score, low content-overlap page count); query result carries a gap flag and targeted ingest suggestions when the wiki lacks relevant coverage; displayed as an Obsidian callout block in the plugin and CLI output\n- **BM25 in-memory corpus cache** — `HybridSearch._cached_corpus` built once per session, invalidated via `invalidate_index()` after each page write; eliminates N×disk reads on decomposed queries\n- **OpenAIProvider contract tests** — 4 tests covering happy path, system message, null content, and custom `base_url` forwarding; applies to OpenAI, Gemini, Groq, and Ollama (all use `OpenAIProvider`)\n- **HTTP 502 on LLM failure** — `/query` GET and POST return 502 Bad Gateway (not raw 500) when the LLM provider is unreachable\n- **Web search decomposition** — `SearchDecomposeAgent` breaks a web search intent into 1–4 focused keyword search strings (separate prompt from query decomposition); parallel Tavily searches; URL deduplication; graceful fallback on LLM error; integrated into `IngestAgent` at the web search fan-out point\n- **New Obsidian commands** — `Lint: run`, `Lint: run with auto-resolve`, `Jobs: retry dead job...`, `Jobs: purge old completed/dead...`, `Wiki: regenerate scaffold...`; audit surfaces added as separate commands (later consolidated into `Audit...` in v0.5.0)\n- **Vector search + semantic re-ranking** — opt-in hybrid BM25 + local vector search using `BAAI/bge-small-en-v1.5` via `fastembed`; one-time background migration embeds existing pages; BM25 serves during migration; enable with `[search] vector = true`\n- **Obsidian web search live view** — `WebSearchModal` replaced with live-polling panel that shows phase text, pages list, and URL errors in real time; configurable poll interval; modal stays open until all fan-out URL jobs settle; job progress tracked via new `progress` column in `jobs.db`\n- **Web search URL cap** — `synthadoc ingest \"search for: …\" --max-results N` limits total URLs enqueued across all sub-queries; Obsidian modal exposes the same as a numeric input (1–50, default 20); cap applied after dedup\n- **Image ingest for OpenAI-compatible providers** — `OpenAIProvider` auto-converts Anthropic image blocks to OpenAI `image_url` format; Groq flagged as non-vision (`supports_vision = False`); image jobs routed to Groq get `fail_permanent` with a clear message\n- **Job crash recovery** — `in_progress` jobs are reset to `pending` on server `init()`, so all pending work resumes automatically after a restart\n- **Rate-limit requeue** — HTTP 429 responses from any LLM provider are detected and requeued via `requeue()` (retry counter unchanged), preserving the retry budget for real errors\n- **Bulk cancel (`jobs cancel`)** — `synthadoc jobs cancel [-w wiki] [--yes]` marks all pending jobs as `skipped` in one operation; also `POST /jobs/cancel-pending`\n\n### v0.3.0 (Community Edition)\n\n- **Session wiki resolution (`synthadoc use`)** — `synthadoc use ` writes the default wiki to `~/.synthadoc/default_wiki`; all commands resolve it automatically via priority chain: `-w` flag > `SYNTHADOC_WIKI` env var > saved default > CWD fallback; hint messages simplified to `[wiki: ]`; `-w .` omitted from job hints when CWD is the active wiki\n- **MiniMax reasoning-model fixes** — `OpenAIProvider` now handles three failure modes of reasoning models (e.g. MiniMax-M2.5): (1) `choices=null` response converted from silent `TypeError` to a descriptive `RuntimeError` with error code logged; (2) `content=null` with prose answer in `reasoning_content` — think-tag stripping then full-text fallback so query synthesis returns a real answer; (3) `APITimeoutError` caught, logged with the config key to set, then re-raised\n- **Configurable LLM call timeout (`agents.llm_timeout_seconds`)** — new `[agents]` key (default `0` = no limit); passed as `timeout` to every OpenAI-compatible `create()` call; `APITimeoutError` logs an actionable message naming the exact config key; config.toml template ships the key commented out with a 5-line explanation of when to enable it\n- **`parse_json_string_array` utility** — extracted shared fence-strip + JSON-parse + filter logic from `QueryAgent.decompose()` and `SearchDecomposeAgent.decompose()` into `synthadoc/agents/_utils.py`; 16 unit tests; LLM call failures and JSON-parse failures now log separate, distinct messages\n- **DeepSeek provider** — `deepseek` added as an eighth provider; routes through `OpenAIProvider` with `base_url=\"https://api.deepseek.com/v1\"` and `DEEPSEEK_API_KEY`; vision disabled (`_NO_VISION_HOSTS`); DeepSeek-R1 `` tags in the `content` field are stripped by the existing regex; config.toml template ships a commented-out example for `deepseek-chat`\n- **YouTube transcript skill** — `synthadoc ingest \"https://www.youtube.com/watch?v=...\"` extracts captions via the YouTube caption system (no API key, no audio download) and feeds the transcript through the existing IngestAgent pipeline. Short videos produce one wiki page; long videos chunk automatically. Graceful skip when no captions are available or the video is private. Tavily web search results that are YouTube URLs are routed automatically via the longest-prefix routing fix in `detect_skill`.\n- **Longest-prefix URL routing** — `detect_skill` now selects the skill whose trigger prefix is the longest match, rather than the first match. This makes YouTube URLs reliably route to the YouTube skill ahead of the generic URL skill, and will correctly handle any future URL-specific skills without priority fields.\n- **v0.2.0 gap fixes** — Ollama `eval_count` mapped to `output_tokens` (was always 0); `_SLUG_BLACKLIST` moved to module-level `frozenset`; synthetic URL fields in ingest_agent commented; four test-coverage gaps closed (no-text guard, orphan flag inversion, `/analyse` endpoint, hybrid-search partial-miss fallback)\n- **Coding tool CLI providers (`claude-code`, `opencode`)** — users with a Claude Code or Opencode subscription can set `provider = \"claude-code\"` (or `\"opencode\"`) in `config.toml` and run all three agents (ingest, query, lint) without a separate API key. `CodingToolCLIProvider` abstract base handles subprocess mechanics (stdin prompt passing, timeout, exit code, stderr capture); `ClaudeCodeCLIProvider` and `OpencodeProvider` each implement `_build_command()`, `_parse_output()`, and `_is_quota_exhausted()`. Quota exhaustion raises `CodingToolQuotaExhaustedException` and permanently fails the job with a clear retry message. `synthadoc serve --provider ` overrides `config.toml` for the lifetime of the server process. Vector search falls back to BM25-only (CLI providers do not support `embed()`). Codex support planned for v0.4.0.\n- **Knowledge gap detection hardening** — signal 5 redesigned from single-discriminating-term check to `min(qualifying_pages per specific term) == 0`, making multi-aspect queries deterministic; Windows asyncio `ConnectionResetError` (WinError 10054) downgraded from ERROR to DEBUG via a scoped exception handler; `aiosqlite` and `asyncio` noisy DEBUG output silenced.\n- **CJK multilingual query support** — Chinese, Japanese, and Korean queries no longer trigger false knowledge-gap reports. `QueryAgent._key_terms` now detects CJK character ranges and skips whitespace-based tokenization (which produces whole-sentence tokens with doc_freq=0), leaving signals 1 and 2 (page count and BM25 score) active for language-agnostic coverage assessment.\n- **ImageSkill standalone refactor** — `ImageSkill` now accepts `provider=` and performs the vision LLM call itself, returning populated text and token counts in `ExtractedContent`. `IngestAgent` injects its provider via `skill_kwargs` (same pattern as `YoutubeSkill`) and no longer contains a special `is_image` branch. The skill is now usable independently of the Synthadoc pipeline.\n- **YouTube executive summary** — each ingested YouTube video page opens with an LLM-generated executive summary (what the video is about, main topics, key takeaway) followed by the full timestamped transcript. Summary is generated once and cached; CJK transcripts receive a higher word-limit for the summary. YouTube Shorts are fully supported alongside standard-length videos.\n- **Obsidian UX improvements** — all modals are draggable and support full text selection and copy-paste; `Lint: run...` consolidates lint and auto-resolve into a single modal with an auto-resolve checkbox; `Jobs: retry failed or dead jobs...` shows a multi-select table with all checkboxes pre-ticked and polls progress live; `Synthadoc: Audit: events...` command added (table of system events with configurable limit); `Ingest: from URL...`, `Ingest: current file`, and `Wiki: regenerate scaffold...` modals all poll job status live.\n\n### v0.4.0 (Community Edition)\n\n- **Routing layer** — `ROUTING.md` groups wiki pages into named topic branches; `QueryAgent` picks 1–2 branches via a lightweight LLM call and restricts BM25 to those slugs, reducing noise on large wikis; falls back to full-corpus search when no branch is selected; `IngestAgent` auto-places new pages into the best branch on create\n- **Alias expansion** — pages may carry `aliases:` in YAML frontmatter; `QueryAgent._expand_aliases` substitutes alias matches in the question with the canonical slug before search (longest-first to avoid partial-match conflicts)\n- **Protected scaffold zone** — `` marker in `index.md` separates user-authored content (preserved) from scaffold-managed content (rewritten); absent marker → full rewrite (original behaviour)\n- **Routing CLI** — `synthadoc routing init / validate / clean` commands manage `ROUTING.md` offline; `init` builds it from the current `index.md` branch structure; `validate` reports dangling slugs; `clean` removes them\n- **Candidates staging** — new pages can be routed to `wiki/candidates/` based on `[ingest] staging_policy` (`off` / `all` / `threshold`); `threshold` mode compares page confidence against `staging_confidence_min`; candidates are excluded from BM25, lint, and contradiction detection until promoted\n- **Candidates CLI** — `synthadoc staging policy` shows/sets the staging policy; `synthadoc candidates list / promote / discard` manage the candidate queue; policy changes take effect on next ingest without a server restart\n- **ContextAgent** — `ContextAgent.build(goal)` decomposes the goal, runs parallel BM25 searches, merges by best score per slug, and greedily packs pages within a configurable token budget; omissions are recorded; output is a `ContextPack` with `to_markdown()` and `to_dict()` renderers\n- **Context CLI + REST endpoint** — `synthadoc context build \"...\"` with `--tokens` and `--output` flags; prints to terminal by default, saves to any file with `--output`; typical uses: paste into an external LLM prompt, save next to a document you are writing, or pipe into another CLI tool; `POST /context/build` JSON endpoint; default token budget configurable via `[query] context_token_budget` (default 4000)\n- **Plugin install CLI** — `synthadoc plugin install ` copies the pre-built Obsidian plugin (`main.js`, `manifest.json`, `styles.css`) from the repo's `obsidian-plugin/` directory into `/.obsidian/plugins/synthadoc/`; replaces the previous manual file-copy step; wiki must be registered via `synthadoc install` first so the path can be resolved from the registry\n- **Plugin upgrade CLI** — `synthadoc plugin upgrade` (no arguments) reads the wiki registry and reinstalls the latest plugin files into every registered vault; run once after each `pip install` upgrade to keep all wikis in sync without having to remember individual `plugin install` calls; wikis with stale registry paths are reported and skipped gracefully\n- **Web search moved into Ingest modal** — the standalone `Synthadoc: Ingest: web search...` command palette entry is removed; web search is now the first tab of `Synthadoc: Ingest...`, consolidating all ingest surfaces in one place\n- **Audit commands consolidated** — the four separate audit command palette entries (`Audit: ingest history...`, `Audit: cost summary...`, `Audit: query history...`, `Audit: events...`) are merged into a single `Synthadoc: Audit...` tabbed modal with tabs for Query history, Ingest history, Events, and Cost summary\n- **Jobs commands consolidated** — the three separate jobs command palette entries (`Jobs: list...`, `Jobs: retry failed or dead jobs...`, `Jobs: purge old completed/dead...`) are merged into a single `Synthadoc: Jobs...` modal; a **Retry selected** button (enabled when ≥ 1 checked job is failed/dead/cancelled) replaces the standalone retry command; a **Purge old jobs** footer row (day threshold input + button) replaces the standalone purge command\n- **ai-research demo template** — `synthadoc install ai-research --target --demo` installs a second demo wiki (alongside history-of-computing) with 12 pre-built AI/ML pages, five raw source files covering multiple ingest scenarios, a contradiction scenario (Gemini Ultra MMLU benchmark methodology dispute), and a pre-configured ROUTING.md\n- **Decision cache prompt-awareness** — the decision-pass cache key (`ck2`) now includes a hash of the full decision prompt (purpose block + instruction template); any change to `purpose.md` content or the purpose-block instructions automatically invalidates cached decisions, preventing stale skip results from being served after prompt edits\n- **YouTube always creates own page** — sources with a structured executive summary (YouTube transcripts) are forced to `action=create` even when the LLM suggests `action=update`, ensuring the executive summary and transcript are never appended to an existing page\n\n### v0.5.0 (Community Edition)\n\n- **Adversarial review** — concurrent independent LLM review of every wiki page after lint runs; flags overstated claims, unsupported assertions, and high-confidence statements the source material does not support; results stored as `lint_warnings: [{claim, concern}]` in page frontmatter; surfaced in redesigned 3-tab `Lint: report` modal (Contradictions / Orphans / Adversarial) and redesigned `synthadoc lint report` CLI output; configured via `[agents].adversarial` and `[lint].adversarial_max_per_page` (default 2) in `config.toml`; skipped with `synthadoc lint run --no-adversarial` (also clears stale warnings); cross-model review — a different model family from the ingest model reduces self-serving bias; concurrent via `asyncio.gather()` — a 100-page wiki completes in the same wall-clock time as one call; per-page rate-limit failures are non-fatal\n- **Claim-level provenance** — during ingest, Pass 4 (`_annotate_citations()`) reads each page section alongside numbered source text and inserts `^[filename:L-L]` inline citation markers at the end of substantive paragraphs; markers map compiled claims to exact source line ranges; stored in the page body, recorded in `audit.db` `claim_citations` table, and validated by lint; local source sidecars written to `.synthadoc/extracted/` (plain-text `.txt` for all file types; pagemap JSON for PDFs to resolve line numbers to PDF page numbers); in Obsidian (Reading View only) markers render as interactive citation chips — one click opens the Source Viewer showing the referenced lines with ±5 lines of context; PDF sources show a page-jump button; `GET /provenance/citations` endpoint powers the **View Page Provenance** modal (sortable, paginated citation table); `synthadoc audit citations` CLI queries the same table with `--page` and `--broken` filters\n- **Routing Obsidian plugin** — `Synthadoc: Routing: manage ROUTING.md...` command palette entry opens a modal panel with three buttons: **Init** creates ROUTING.md from the current index.md branch structure (enabled only when ROUTING.md does not exist), **Validate** reports dangling slugs, **Clean** removes dangling slugs from ROUTING.md; after each action results appear inline\n- **Candidates Staging Obsidian plugin** — `Synthadoc: Staging: manage staging policy...` and `Synthadoc: Candidates: review candidate pages...` command palette entries; Staging modal shows policy state with segmented controls; Candidates modal shows a paginated table with promote/discard bulk and per-row actions\n### v0.6.0 (Community Edition)\n\n- **5-state lifecycle machine** — every wiki page tracks a `draft | active | contradicted | stale | archived` state in two new `audit.db` tables: `page_states` (fast current-state index, slug PK) and `lifecycle_events` (immutable audit log of every transition with slug, from/to state, reason, triggered_by, and timestamp)\n- **Ingest creates draft pages** — all new pages are created with `status: draft` instead of `active`; pages must pass a lint run to be promoted\n- **LintAgent lifecycle checks** — four automated checks run at the end of every lint pass: archived detection (source file missing → `archived`), stale detection (source hash mismatch → `stale`), draft promotion (draft + no active issues → `active`), manual-edit sync (frontmatter `status` ≠ DB → reconcile); skipped when `--no-lifecycle` is passed\n- **Auto-retention** — `[audit] lifecycle_retention_days = N` in `config.toml` prunes old `lifecycle_events` at the end of each lint run; `0` = keep forever (default)\n- **Lifecycle CLI** — `synthadoc lifecycle activate/archive/restore/log`, `synthadoc status` extended with per-state counts, `synthadoc audit lifecycle purge --before / --keep-latest`\n- **Lifecycle HTTP API** — `GET /lifecycle/status`, `GET /lifecycle/events`, `POST /lifecycle/transition`\n- **Lifecycle Obsidian plugin** — `Synthadoc: Manage Page Lifecycle` command opens `LifecycleModal`: sortable, filterable, paginated table of all pages with current state and last transition; valid transition action buttons per row; `ReasonModal` prompts for reason before committing; draft/stale badge links on lint modal and jobs panel open the table pre-filtered\n- **Export formats** — `synthadoc export --format ` serializes the wiki in four formats assembled server-side with zero LLM calls: `llms.txt` (navigation index per llmstxt.org spec — active pages in `## Pages`, contradicted/stale in `## Needs Review`, archived omitted); `llms-full.txt` (flat content dump with `---` separators, provenance footnotes preserved verbatim, no size limit); `graphml` (standard GraphML 1.1 — node attributes include `label`/`title`, `status`, `confidence`, `orphan`, `inbound_link_count`, `routing_branch`; edges=wikilinks; dual-label support: `label` key for Gephi/Cytoscape, `y:NodeLabel` for yEd; no position data — run tool layout after import); `json` (agent-ready dump with `claims[]`, `lifecycle_history[]`, per-page `ingest_cost_usd` and `ingest_tokens`, `total_compilation_cost_usd`, `routing.branch_memberships`); all formats accept `--status` filter (`all`/`active`/`draft`/`stale`/`contradicted`/`archived`); `POST /export` endpoint accepts `{format, status_filter}`; Obsidian **Export Wiki** command — format dropdown, full-width output path, status filter, Export button, View Graph inline preview button (graphml only)\n\n\n### v0.7.0 (Community Edition)\n\n- **Streaming query output** — `synthadoc query` streams the LLM answer token-by-token via Server-Sent Events (SSE); the CLI renders tokens as they arrive. `--no-stream` reverts to blocking mode for scripts and pipes. `GET /query/stream` SSE endpoint; the streaming path reuses all existing decomposition, BM25, and knowledge-gap logic.\n- **Epoch-based query result cache** — query answers are cached in `cache.db` under a composite key of normalised question + wiki epoch + cache version. The wiki epoch increments on every ingest completion and every lifecycle transition, so cached answers are always consistent with current wiki content. `--no-cache` bypasses the cache per call without evicting the entry. `synthadoc cache clear` resets the epoch and removes all entries.\n- **Web chat UI** — React SPA served at `GET /app` by the existing HTTP server (no extra port or process). Features: session-aware mode detection (`NEW_WIKI` / `EXPLORER` / `HEALTH_CHECK` / `POWER_USER`) based on wiki page count, prior session history, and stale-page presence; streaming answers via SSE; citation links; knowledge-gap callouts; contextual hint chips with windowed pool rotation; and multi-turn conversation history. Server maintains a lightweight in-memory session store (session_id, mode, hint cursor, last_hints).\n- **Session management and hint engine** — hint chip sets are derived from wiki index headings (new mode), recent citation graph (explorer mode), and uncovered BM25 top pages (power-user mode). Mode badge displayed in the UI header.\n- **Obsidian plugin streaming** — the Query command streams the answer into the modal as tokens arrive, matching the CLI and web UI streaming experience. No change to the Obsidian command count.\n- **`synthadoc web` CLI command** — opens the default browser to `http://localhost:{port}/app`; thin wrapper around OS open/start/xdg-open; server must already be running. The web UI is local-only; network access and authentication are not available in the Community Edition.\n- **Action Agent** — regex pre-filter + LLM extraction layer that detects action-intent queries (\"run lint\", \"schedule ingest every night\", \"what pages are orphans?\") and dispatches them to live Synthadoc operations without going through the query pipeline. Supports: `lint`, `lint_report`, `wiki_status`, `ingest`, `scaffold`, `schedule_add`, `schedule_list`, `schedule_history`, `lifecycle_activate`, `lifecycle_archive`, `lifecycle_restore`. Returns structured `ActionResult` rendered directly in the web UI and CLI.\n- **Expanded hint coverage** — built-in `hints.json` extended with lifecycle action hints (\"Activate a draft page\", \"Archive a stale page\", \"Restore an archived page to draft\"), orphan/contradiction/adversarial patterns, and \"Show wiki status\" in EXPLORER and HEALTH_CHECK modes.\n\n### v0.8.0 (Community Edition)\n\n- **Multi-turn conversation** — the web chat UI maintains conversation history across turns within a session. History is stored in `audit.db` per session and loaded server-side on each request (up to `conversation_history_turns` turns, default 10). Follow-up questions are rewritten into standalone form by a dedicated rewrite component before BM25 retrieval, so context-dependent phrases (\"What came after that?\") resolve correctly. When the session exceeds the turn limit, a summarization component compresses the oldest turns into a `[Session summary]` entry; a `notice` SSE event is emitted the first time compression occurs.\n- **Clarify event** — when an action-intent query is ambiguous (e.g. \"activate a draft page\" without specifying which page), the server emits a `clarify` SSE event with a disambiguation prompt and candidate page list instead of routing to the synthesis pipeline. The web UI renders candidates as numbered chip buttons; the user can tap a chip or type a page name to complete the action.\n- **Two new SSE events** — `clarify` (`{prompt, candidates, action}`) for action disambiguation; `notice` (`{text}`) for system messages such as history compression.\n- **Configuration** — `[chat] conversation_history_turns = 5` controls how many prior turns are included in each request. Set to `0` to disable conversation history. `clarify_lookback = 5` controls how many prior assistant turns to scan when detecting a clarify continuation (chip click after an ambiguous action query); configurable independently of `conversation_history_turns`.\n- **MiniMax M3 support** — `provider = \"minimax\"` with `model = \"MiniMax-Text-01\"` or `\"MiniMax-M3\"` (thinking mode configurable; off by default).\n- **Settings gear** — web UI chat window now has a ⚙ gear button that opens a popover to configure the per-request query timeout (10–600 s, default 60 s). Value persisted in `localStorage`.\n- **Query timeout** — `GET /query/stream` accepts `?timeout_seconds=N`; a `TimeoutError` emits an SSE `error` event to the browser.\n- **Session history sidebar** — the left navigation bar is now server-driven (`GET /sessions`) and shows sessions as a 2-level collapsible tree. Multi-turn sessions display a turn count badge and expand to show each follow-up question. Clicking any item restores the full conversation history via `GET /sessions/{id}/messages`. Two new API endpoints: `GET /sessions` and `GET /sessions/{session_id}/messages`.\n- **Job status and list actions** — the Action Agent now handles `job_status` and `job_list` intent queries. `job_status` with a job ID returns a detailed job card; without an ID it returns a table of all jobs and emits a `clarify` event so the user can pick one via chip. `job_list` accepts an optional multi-status filter (e.g. \"show failed and skipped jobs\") and includes an Error column when any listed job has a non-null error. Built-in `hints.json` extended with job-status and job-list hints for POWER_USER mode.\n- **Multi-chip clarify continuation** — clarify chip replies (bare UUIDs) are now reliably routed back to the Action Agent across multiple chip clicks. The server tags every clarify message with a `[clarify] ` prefix in the audit log; `detect()` scans back `clarify_lookback` assistant turns to find an open clarify context.\n- **Qwen provider routing** — `qwen-` model names (e.g. `qwen-plus`, `qwen-max`) route to DashScope cloud API regardless of other config; all other Qwen models (e.g. `qwen3:8b`, `qwen3.5`) route to local Ollama. This decouples cloud/local routing from `QWEN_API_KEY` presence.\n"}}}}