{"repo": "chaelsoo/zetsu", "files": {"README.md": {"de71caf42e56f9fec926fc6ca25195f055da09f0": {"t": 1785746365.2957976, "n": 6824, "h": "4aff199d16caaa69", "c": "

\n \"ZETSU\"\n

\n\nA personal offline RAG system for offensive security knowledge. Ingest your writeups, lab solutions, bug bounty reports, and security blogs, then query them in natural language during engagements. No cloud, no SaaS, your knowledge stays yours.\n\n\n## What it does\n\nZETSU turns your accumulated knowledge into a queryable assistant. Instead of grepping through markdown files or remembering which writeup had that certipy command, you ask naturally:\n\n```\nhow do i escalate with SeImpersonatePrivilege\nwhat did i do after getting ADFS access\nsliver socks5 pivot setup\nexplain ESC8 vs ESC4\n```\n\nIt retrieves from your actual notes first, then generates an answer grounded in what you've documented, not generic internet knowledge.\n\n\n## Architecture\n\n### Ingestion pipeline\n\n![Ingestion pipeline](assets/ingestion.png)\n\n### Query pipeline\n\n![Query pipeline](assets/query.png)\n\n\n## Features\n\n- **FARR extraction:** at ingest time, an LLM reads each section of your writeups and extracts structured attack steps (Finding, Action, Reasoning, Result). What you retrieve is a semantic unit, not a random 800-token window.\n- **Hybrid retrieval:** BM25 for exact token matching (tool names, CVE numbers, CLI flags) + vector search for semantic similarity. RRF fusion combines both.\n- **Cross-encoder reranker:** reorders retrieved chunks by actual relevance before sending to the LLM.\n- **Two response styles:** Operator mode leads with exact commands, Concept mode leads with reasoning. Same retrieval, different presentation. Toggle with `Ctrl+M` in TUI.\n- **Multiple LLM backends:** Anthropic, OpenAI-compatible (DeepSeek, etc.), or local Ollama. Separate backends for ingest and query.\n- **Multiple source types:** local markdown files, single URLs, GitHub wikis, Atom/RSS feeds.\n- **Persistent chat history:** saved to `~/.zetsu/history/YYYY-MM-DD.json`, browsable with `/history`.\n- **TUI + Web UI + CLI:** use what fits the context. TUI during engagements, Web UI for studying, CLI for scripting.\n\n\n## Installation\n\n```bash\ngit clone https://github.com/chaelsoo/zetsu\ncd zetsu\npip install -r requirements.txt\n```\n\nSet up your API key:\n\n```bash\ncp .env.example .env\n# edit .env and add your key\n```\n\n\n## Quick start\n\n**1. Add your writeups**\n\nDrop `.md` files into `docs/`. Notion exports, Obsidian notes, blog exports, anything markdown works.\n\n**2. Configure sources in `config.toml`**\n\n```toml\n[llm]\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\n\n[[sources]]\ntype = \"markdown_dir\"\npath = \"./docs\"\nname = \"my_writeups\"\nextract = \"farr\"\nenabled = true\n```\n\n**3. Ingest**\n\n```bash\npython zetsu.py ingest\n```\n\n**4. Use it**\n\n```bash\npython zetsu.py tui # terminal UI\npython zetsu.py web # browser at localhost:8000\npython zetsu.py ask \"how do i abuse SeImpersonatePrivilege\"\n```\n\n\n## CLI reference\n\n```bash\npython zetsu.py ingest # ingest enabled sources\npython zetsu.py ingest --force # wipe and rebuild everything\npython zetsu.py ingest --dry-run # estimate without making LLM calls\npython zetsu.py ingest --all-sources # enable all sources regardless of enabled flag\npython zetsu.py tui # terminal UI\npython zetsu.py web # web UI\npython zetsu.py ask \"query\" # one-shot CLI\npython zetsu.py ask \"query\" --style concept\npython zetsu.py stats # vector store stats\n```\n\n\n## TUI keybindings\n\n| Key | Action |\n|-----|--------|\n| `Enter` | Send query |\n| `Ctrl+M` | Toggle Operator / Concept mode |\n| `Ctrl+Y` | Copy last response to clipboard |\n| `Ctrl+L` | Clear session |\n| `Ctrl+C` | Quit |\n| `/help` | Show commands |\n| `/stats` | Vector store + history stats |\n| `/history` | Today's past queries |\n| `/clear` | Clear session |\n\n\n## Source types\n\n| Type | Use for | Extract mode |\n|------|---------|--------------|\n| `markdown_dir` | Your writeups, notes | `farr` |\n| `markdown_file` | Single markdown file | `farr` |\n| `url` | Tool wikis, reference pages | `headers` |\n| `atom` | Security blogs with feeds | `farr+narrative` |\n\n### Extract modes\n\n- **`farr`:** LLM extracts discrete attack steps as structured JSON (Finding, Action, Reasoning, Result). Best for writeups with clear attack chains.\n- **`farr+narrative`:** same as farr, plus a prose summary capturing the author's reasoning. Best for blogs where the thought process matters.\n- **`headers`:** no LLM, just split on headers and keep verbatim. Best for command references and tool documentation.\n\n---\n\n## LLM backends\n\nConfigure in `config.toml`. Separate backends for ingestion (bulk extraction) and queries (generation):\n\n```toml\n[llm] # query time\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\n\n[ingest] # ingest time\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\nworkers = 8 # parallel files during FARR extraction\n```\n\nSupported backends: `anthropic`, `openai` (any OpenAI-compatible API), `ollama`.\n\n\n> [!IMPORTANT]\n> ChromaDB is run locally by default, which loads the full vector index into RAM. For large corpora (10k+ chunks) or memory-constrained machines, it is strongly recommended to run ChromaDB as a remote server instead. See [ChromaDB server docs](https://docs.trychroma.com/production/chroma-server/client-server-mode) for setup. Remote support in ZETSU is planned.\n\n## Benchmark\n\nEvaluated across 910 questions covering 12 offensive security categories\n(ADCS, Kerberos, AD enumeration, MSSQL, Sliver C2, privilege escalation,\nweb attacks, credentials, lateral movement, cloud/Entra ID, OPSEC, tools).\n\n| Metric | Result |\n|--------|--------|\n| Questions answered | 910 / 910 (100%) |\n| Answers with code/commands | 842 / 910 (93%) |\n| Context gaps (model admitted missing info) | 66 / 910 (7.3%) |\n| Avg retrieval time | 68ms |\n| Avg total response time | 3.8s |\n\nTop sources contributing to answers: personal HTB writeups, dirkjanm blog, shenaniganslabs research, HackTricks ADCS, Sliver wiki, netexec cheatsheet.\n\nIn simple words, you need to make sure to add good & structured resources that you want the RAG to ingest, this project focueses 100% on using a knowledge base, not trusting the LLM's memory or trained on knowledge.\n\nFull evaluation dataset and results in [`eval/`](eval/).\n\n## References\n\n- [rank-bm25](https://github.com/dorianbrown/rank_bm25), BM25 retrieval\n- FARR concept from [CIPHER](https://github.com/ibndias/CIPHER)\n- RAG pipeline inspired by [rag-chatbot](https://github.com/umbertogriffo/rag-chatbot)\n\n> I will be documenting in a blog the usage and how effective it is very soon, stay tuned.\n"}, "dc44ab99b6c1384d9cc1572b92aa94940e64858b": {"t": 1785762661.7486327, "n": 5900, "h": "6cea9ab4e9ac086a", "c": "

\n \"ZETSU\"\n

\n\nA personal offline RAG system for offensive security knowledge. Ingest your writeups, lab solutions, bug bounty reports, and security blogs, then query them in natural language during engagements. No cloud, no SaaS, your knowledge stays yours.\n\n\n## What it does\n\nZETSU turns your accumulated knowledge into a queryable assistant. Instead of grepping through markdown files or remembering which writeup had that certipy command, you ask naturally:\n\n```\nhow do i escalate with SeImpersonatePrivilege\nwhat did i do after getting ADFS access\nsliver socks5 pivot setup\nexplain ESC8 vs ESC4\n```\n\nIt retrieves from your actual notes first, then generates an answer grounded in what you've documented, not generic internet knowledge.\n\n\n## Architecture\n\n### Ingestion pipeline\n\n![Ingestion pipeline](assets/ingestion.png)\n\n### Query pipeline\n\n![Query pipeline](assets/query.png)\n\n\n## Features\n\n- **FARR extraction:** at ingest time, an LLM reads each section of your writeups and extracts structured attack steps (Finding, Action, Reasoning, Result). What you retrieve is a semantic unit, not a random 800-token window.\n- **Hybrid retrieval:** BM25 for exact token matching (tool names, CVE numbers, CLI flags) + vector search for semantic similarity. RRF fusion combines both.\n- **Cross-encoder reranker:** reorders retrieved chunks by actual relevance before sending to the LLM.\n- **Two response styles:** Operator mode leads with exact commands, Concept mode leads with reasoning. Same retrieval, different presentation. Toggle with `Ctrl+M` in TUI.\n- **Multiple LLM backends:** Anthropic, OpenAI-compatible (DeepSeek, etc.), or local Ollama. Separate backends for ingest and query.\n- **Multiple source types:** local markdown files, single URLs, GitHub wikis, Atom/RSS feeds.\n- **Persistent chat history:** saved to `~/.zetsu/history/YYYY-MM-DD.json`, browsable with `/history`.\n- **TUI + Web UI + CLI:** use what fits the context. TUI during engagements, Web UI for studying, CLI for scripting.\n\n\n## Installation\n\n```bash\ngit clone https://github.com/chaelsoo/zetsu\ncd zetsu\npip install -r requirements.txt\n```\n\nSet up your API key:\n\n```bash\ncp .env.example .env\n# edit .env and add your key\n```\n\n\n## Quick start\n\n**1. Add your writeups**\n\nDrop `.md` files into `docs/`. Notion exports, Obsidian notes, blog exports, anything markdown works.\n\n**2. Configure sources in `config.toml`**\n\n```toml\n[llm]\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\n\n[[sources]]\ntype = \"markdown_dir\"\npath = \"./docs\"\nname = \"my_writeups\"\nextract = \"farr\"\nenabled = true\n```\n\n**3. Ingest**\n\n```bash\npython zetsu.py ingest\n```\n\n**4. Use it**\n\n```bash\npython zetsu.py tui # terminal UI\npython zetsu.py web # browser at localhost:8000\npython zetsu.py ask \"how do i abuse SeImpersonatePrivilege\"\n```\n\n\n## CLI reference\n\n```bash\npython zetsu.py ingest # ingest enabled sources\npython zetsu.py ingest --force # wipe and rebuild everything\npython zetsu.py ingest --dry-run # estimate without making LLM calls\npython zetsu.py ingest --all-sources # enable all sources regardless of enabled flag\npython zetsu.py tui # terminal UI\npython zetsu.py web # web UI\npython zetsu.py ask \"query\" # one-shot CLI\npython zetsu.py ask \"query\" --style concept\npython zetsu.py stats # vector store stats\n```\n\n\n## TUI keybindings\n\n| Key | Action |\n|-----|--------|\n| `Enter` | Send query |\n| `Ctrl+M` | Toggle Operator / Concept mode |\n| `Ctrl+Y` | Copy last response to clipboard |\n| `Ctrl+L` | Clear session |\n| `Ctrl+C` | Quit |\n| `/help` | Show commands |\n| `/stats` | Vector store + history stats |\n| `/history` | Today's past queries |\n| `/clear` | Clear session |\n\n\n## Source types\n\n| Type | Use for | Extract mode |\n|------|---------|--------------|\n| `markdown_dir` | Your writeups, notes | `farr` |\n| `markdown_file` | Single markdown file | `farr` |\n| `url` | Tool wikis, reference pages | `headers` |\n| `atom` | Security blogs with feeds | `farr+narrative` |\n\n### Extract modes\n\n- **`farr`:** LLM extracts discrete attack steps as structured JSON (Finding, Action, Reasoning, Result). Best for writeups with clear attack chains.\n- **`farr+narrative`:** same as farr, plus a prose summary capturing the author's reasoning. Best for blogs where the thought process matters.\n- **`headers`:** no LLM, just split on headers and keep verbatim. Best for command references and tool documentation.\n\n---\n\n## LLM backends\n\nConfigure in `config.toml`. Separate backends for ingestion (bulk extraction) and queries (generation):\n\n```toml\n[llm] # query time\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\n\n[ingest] # ingest time\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\nworkers = 8 # parallel files during FARR extraction\n```\n\nSupported backends: `anthropic`, `openai` (any OpenAI-compatible API), `ollama`.\n\n\n> [!IMPORTANT]\n> ChromaDB is run locally by default, which loads the full vector index into RAM. For large corpora (10k+ chunks) or memory-constrained machines, it is strongly recommended to run ChromaDB as a remote server instead. See [ChromaDB server docs](https://docs.trychroma.com/production/chroma-server/client-server-mode) for setup. Remote support in ZETSU is planned.\n\n\n## References\n\n- [rank-bm25](https://github.com/dorianbrown/rank_bm25), BM25 retrieval\n- FARR concept from [CIPHER](https://github.com/ibndias/CIPHER)\n- RAG pipeline inspired by [rag-chatbot](https://github.com/umbertogriffo/rag-chatbot)\n\n> I will be documenting in a blog the usage and how effective it is very soon, stay tuned.\n"}, "753f6d3972a9abea0ff640a4fad206f527f5a8b4": {"t": 1785762665.6965718, "n": 5924, "h": "e515d46bf6c3a86d", "c": "

\n \"ZETSU\"\n

\n\nA personal offline RAG system for offensive security knowledge. Ingest your writeups, lab solutions, bug bounty reports, and security blogs, then query them in natural language during engagements. No cloud, no SaaS, your knowledge stays yours.\n\n\n## What it does\n\nZETSU turns your accumulated knowledge into a queryable assistant. Instead of grepping through markdown files or remembering which writeup had that certipy command, you ask naturally:\n\n```\nhow do i escalate with SeImpersonatePrivilege\nwhat did i do after getting ADFS access\nsliver socks5 pivot setup\nexplain ESC8 vs ESC4\n```\n\nIt retrieves from your actual notes first, then generates an answer grounded in what you've documented, not generic internet knowledge.\n\n\n## Architecture\n\n### Ingestion pipeline\n\n![Ingestion pipeline](assets/ingestion.png)\n\n### Query pipeline\n\n![Query pipeline](assets/query.png)\n\n\n## Features\n\n- **FARR extraction:** at ingest time, an LLM reads each section of your writeups and extracts structured attack steps (Finding, Action, Reasoning, Result). What you retrieve is a semantic unit, not a random 800-token window.\n- **Hybrid retrieval:** BM25 for exact token matching (tool names, CVE numbers, CLI flags) + vector search for semantic similarity. RRF fusion combines both.\n- **Cross-encoder reranker:** reorders retrieved chunks by actual relevance before sending to the LLM.\n- **Two response styles:** Operator mode leads with exact commands, Concept mode leads with reasoning. Same retrieval, different presentation. Toggle with `Ctrl+M` in TUI.\n- **Multiple LLM backends:** Anthropic, OpenAI-compatible (DeepSeek, etc.), or local Ollama. Separate backends for ingest and query.\n- **Multiple source types:** local markdown files, single URLs, GitHub wikis, Atom/RSS feeds.\n- **Persistent chat history:** saved to `~/.zetsu/history/YYYY-MM-DD.json`, browsable with `/history`.\n- **TUI + Web UI + CLI:** use what fits the context. TUI during engagements, Web UI for studying, CLI for scripting.\n\n---\n\n## Installation\n\n```bash\ngit clone https://github.com/chaelsoo/zetsu\ncd zetsu\npip install -r requirements.txt\n```\n\nSet up your API key:\n\n```bash\ncp .env.example .env\n# edit .env and add your key\n```\n\n---\n\n## Quick start\n\n**1. Add your writeups**\n\nDrop `.md` files into `docs/`. Notion exports, Obsidian notes, blog exports, anything markdown works.\n\n**2. Configure sources in `config.toml`**\n\n```toml\n[llm]\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\n\n[[sources]]\ntype = \"markdown_dir\"\npath = \"./docs\"\nname = \"my_writeups\"\nextract = \"farr\"\nenabled = true\n```\n\n**3. Ingest**\n\n```bash\npython zetsu.py ingest\n```\n\n**4. Use it**\n\n```bash\npython zetsu.py tui # terminal UI\npython zetsu.py web # browser at localhost:8000\npython zetsu.py ask \"how do i abuse SeImpersonatePrivilege\"\n```\n\n---\n\n## CLI reference\n\n```bash\npython zetsu.py ingest # ingest enabled sources\npython zetsu.py ingest --force # wipe and rebuild everything\npython zetsu.py ingest --dry-run # estimate without making LLM calls\npython zetsu.py ingest --all-sources # enable all sources regardless of enabled flag\npython zetsu.py tui # terminal UI\npython zetsu.py web # web UI\npython zetsu.py ask \"query\" # one-shot CLI\npython zetsu.py ask \"query\" --style concept\npython zetsu.py stats # vector store stats\n```\n\n---\n\n## TUI keybindings\n\n| Key | Action |\n|-----|--------|\n| `Enter` | Send query |\n| `Ctrl+M` | Toggle Operator / Concept mode |\n| `Ctrl+Y` | Copy last response to clipboard |\n| `Ctrl+L` | Clear session |\n| `Ctrl+C` | Quit |\n| `/help` | Show commands |\n| `/stats` | Vector store + history stats |\n| `/history` | Today's past queries |\n| `/clear` | Clear session |\n\n---\n\n## Source types\n\n| Type | Use for | Extract mode |\n|------|---------|--------------|\n| `markdown_dir` | Your writeups, notes | `farr` |\n| `markdown_file` | Single markdown file | `farr` |\n| `url` | Tool wikis, reference pages | `headers` |\n| `atom` | Security blogs with feeds | `farr+narrative` |\n\n### Extract modes\n\n- **`farr`:** LLM extracts discrete attack steps as structured JSON (Finding, Action, Reasoning, Result). Best for writeups with clear attack chains.\n- **`farr+narrative`:** same as farr, plus a prose summary capturing the author's reasoning. Best for blogs where the thought process matters.\n- **`headers`:** no LLM, just split on headers and keep verbatim. Best for command references and tool documentation.\n\n---\n\n## LLM backends\n\nConfigure in `config.toml`. Separate backends for ingestion (bulk extraction) and queries (generation):\n\n```toml\n[llm] # query time\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\n\n[ingest] # ingest time\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\nworkers = 8 # parallel files during FARR extraction\n```\n\nSupported backends: `anthropic`, `openai` (any OpenAI-compatible API), `ollama`.\n\n---\n\n> [!IMPORTANT]\n> ChromaDB is run locally by default, which loads the full vector index into RAM. For large corpora (10k+ chunks) or memory-constrained machines, it is strongly recommended to run ChromaDB as a remote server instead. See [ChromaDB server docs](https://docs.trychroma.com/production/chroma-server/client-server-mode) for setup. Remote support in ZETSU is planned.\n\n\n## References\n\n- [rank-bm25](https://github.com/dorianbrown/rank_bm25), BM25 retrieval\n- FARR concept from [CIPHER](https://github.com/ibndias/CIPHER)\n- RAG pipeline inspired by [rag-chatbot](https://github.com/umbertogriffo/rag-chatbot)\n\n> I will be documenting in a blog the usage and how effective it is very soon, stay tuned.\n"}, "9cf3cf5c8272402bfd96e47a042b589674b3fc15": {"t": 1785762668.1085274, "n": 5832, "h": "1aef1e8d03a2486a", "c": "

\n \"ZETSU\"\n

\n\nA personal offline RAG system for offensive security knowledge. Ingest your writeups, lab solutions, bug bounty reports, and security blogs, then query them in natural language during engagements. No cloud, no SaaS, your knowledge stays yours.\n\n\n## What it does\n\nZETSU turns your accumulated knowledge into a queryable assistant. Instead of grepping through markdown files or remembering which writeup had that certipy command, you ask naturally:\n\n```\nhow do i escalate with SeImpersonatePrivilege\nwhat did i do after getting ADFS access\nsliver socks5 pivot setup\nexplain ESC8 vs ESC4\n```\n\nIt retrieves from your actual notes first, then generates an answer grounded in what you've documented, not generic internet knowledge.\n\n\n## Architecture\n\n### Ingestion pipeline\n\n![Ingestion pipeline](assets/ingestion.png)\n\n### Query pipeline\n\n![Query pipeline](assets/query.png)\n\n\n## Features\n\n- **FARR extraction:** at ingest time, an LLM reads each section of your writeups and extracts structured attack steps (Finding, Action, Reasoning, Result). What you retrieve is a semantic unit, not a random 800-token window.\n- **Hybrid retrieval:** BM25 for exact token matching (tool names, CVE numbers, CLI flags) + vector search for semantic similarity. RRF fusion combines both.\n- **Cross-encoder reranker:** reorders retrieved chunks by actual relevance before sending to the LLM.\n- **Two response styles:** Operator mode leads with exact commands, Concept mode leads with reasoning. Same retrieval, different presentation. Toggle with `Ctrl+M` in TUI.\n- **Multiple LLM backends:** Anthropic, OpenAI-compatible (DeepSeek, etc.), or local Ollama. Separate backends for ingest and query.\n- **Multiple source types:** local markdown files, single URLs, GitHub wikis, Atom/RSS feeds.\n- **Persistent chat history:** saved to `~/.zetsu/history/YYYY-MM-DD.json`, browsable with `/history`.\n- **TUI + Web UI + CLI:** use what fits the context. TUI during engagements, Web UI for studying, CLI for scripting.\n\n---\n\n## Installation\n\n```bash\ngit clone https://github.com/chaelsoo/zetsu\ncd zetsu\npip install -r requirements.txt\n```\n\nSet up your API key:\n\n```bash\ncp .env.example .env\n# edit .env and add your key\n```\n\n---\n\n## Quick start\n\n**1. Add your writeups**\n\nDrop `.md` files into `docs/`. Notion exports, Obsidian notes, blog exports, anything markdown works.\n\n**2. Configure sources in `config.toml`**\n\n```toml\n[llm]\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\n\n[[sources]]\ntype = \"markdown_dir\"\npath = \"./docs\"\nname = \"my_writeups\"\nextract = \"farr\"\nenabled = true\n```\n\n**3. Ingest**\n\n```bash\npython zetsu.py ingest\n```\n\n**4. Use it**\n\n```bash\npython zetsu.py tui # terminal UI\npython zetsu.py web # browser at localhost:8000\npython zetsu.py ask \"how do i abuse SeImpersonatePrivilege\"\n```\n\n---\n\n## CLI reference\n\n```bash\npython zetsu.py ingest # ingest enabled sources\npython zetsu.py ingest --force # wipe and rebuild everything\npython zetsu.py ingest --dry-run # estimate without making LLM calls\npython zetsu.py ingest --all-sources # enable all sources regardless of enabled flag\npython zetsu.py tui # terminal UI\npython zetsu.py web # web UI\npython zetsu.py ask \"query\" # one-shot CLI\npython zetsu.py ask \"query\" --style concept\npython zetsu.py stats # vector store stats\n```\n\n---\n\n## TUI keybindings\n\n| Key | Action |\n|-----|--------|\n| `Enter` | Send query |\n| `Ctrl+M` | Toggle Operator / Concept mode |\n| `Ctrl+Y` | Copy last response to clipboard |\n| `Ctrl+L` | Clear session |\n| `Ctrl+C` | Quit |\n| `/help` | Show commands |\n| `/stats` | Vector store + history stats |\n| `/history` | Today's past queries |\n| `/clear` | Clear session |\n\n---\n\n## Source types\n\n| Type | Use for | Extract mode |\n|------|---------|--------------|\n| `markdown_dir` | Your writeups, notes | `farr` |\n| `markdown_file` | Single markdown file | `farr` |\n| `url` | Tool wikis, reference pages | `headers` |\n| `atom` | Security blogs with feeds | `farr+narrative` |\n\n### Extract modes\n\n- **`farr`:** LLM extracts discrete attack steps as structured JSON (Finding, Action, Reasoning, Result). Best for writeups with clear attack chains.\n- **`farr+narrative`:** same as farr, plus a prose summary capturing the author's reasoning. Best for blogs where the thought process matters.\n- **`headers`:** no LLM, just split on headers and keep verbatim. Best for command references and tool documentation.\n\n---\n\n## LLM backends\n\nConfigure in `config.toml`. Separate backends for ingestion (bulk extraction) and queries (generation):\n\n```toml\n[llm] # query time\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\n\n[ingest] # ingest time\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\nworkers = 8 # parallel files during FARR extraction\n```\n\nSupported backends: `anthropic`, `openai` (any OpenAI-compatible API), `ollama`.\n\n---\n\n> [!IMPORTANT]\n> ChromaDB is run locally by default, which loads the full vector index into RAM. For large corpora (10k+ chunks) or memory-constrained machines, it is strongly recommended to run ChromaDB as a remote server instead. See [ChromaDB server docs](https://docs.trychroma.com/production/chroma-server/client-server-mode) for setup. Remote support in ZETSU is planned.\n\n\n## References\n\n- [rank-bm25](https://github.com/dorianbrown/rank_bm25), BM25 retrieval\n- FARR concept from [CIPHER](https://github.com/ibndias/CIPHER)\n- RAG pipeline inspired by [rag-chatbot](https://github.com/umbertogriffo/rag-chatbot)\n"}, "a4800107aa4267829bf4cdac204d145decc81b7e": {"t": 1785762669.4824939, "n": 5962, "h": "b424fdd283196743", "c": "

\n \"ZETSU\"\n

\n\nA personal offline RAG system for offensive security knowledge. Ingest your writeups, lab solutions, bug bounty reports, and security blogs, then query them in natural language during engagements. No cloud, no SaaS, your knowledge stays yours.\n\n\n---\n\n## What it does\n\nZETSU turns your accumulated knowledge into a queryable assistant. Instead of grepping through markdown files or remembering which writeup had that certipy command, you ask naturally:\n\n```\nhow do i escalate with SeImpersonatePrivilege\nwhat did i do after getting ADFS access\nsliver socks5 pivot setup\nexplain ESC8 vs ESC4\n```\n\nIt retrieves from your actual notes first, then generates an answer grounded in what you've documented, not generic internet knowledge.\n\n---\n\n## Architecture\n\n### Ingestion pipeline\n\n![Ingestion pipeline](assets/ingestion.png)\n\n### Query pipeline\n\n![Query pipeline](assets/query.png)\n\n---\n\n## Features\n\n- **FARR extraction:** at ingest time, an LLM reads each section of your writeups and extracts structured attack steps (Finding, Action, Reasoning, Result). What you retrieve is a semantic unit, not a random 800-token window.\n- **Hybrid retrieval:** BM25 for exact token matching (tool names, CVE numbers, CLI flags) + vector search for semantic similarity. RRF fusion combines both.\n- **Cross-encoder reranker:** reorders retrieved chunks by actual relevance before sending to the LLM.\n- **Two response styles:** Operator mode leads with exact commands, Concept mode leads with reasoning. Same retrieval, different presentation. Toggle with `Ctrl+M` in TUI.\n- **Multiple LLM backends:** Anthropic, OpenAI-compatible (DeepSeek, etc.), or local Ollama. Separate backends for ingest and query.\n- **Multiple source types:** local markdown files, single URLs, GitHub wikis, Atom/RSS feeds.\n- **Persistent chat history:** saved to `~/.zetsu/history/YYYY-MM-DD.json`, browsable with `/history`.\n- **TUI + Web UI + CLI:** use what fits the context. TUI during engagements, Web UI for studying, CLI for scripting.\n\n---\n\n## Installation\n\n```bash\ngit clone https://github.com/chaelsoo/zetsu\ncd zetsu\npip install -r requirements.txt\n```\n\nSet up your API key:\n\n```bash\ncp .env.example .env\n# edit .env and add your key\n```\n\n---\n\n## Quick start\n\n**1. Add your writeups**\n\nDrop `.md` files into `docs/`. Notion exports, Obsidian notes, blog exports, anything markdown works.\n\n**2. Configure sources in `config.toml`**\n\n```toml\n[llm]\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\n\n[[sources]]\ntype = \"markdown_dir\"\npath = \"./docs\"\nname = \"my_writeups\"\nextract = \"farr\"\nenabled = true\n```\n\n**3. Ingest**\n\n```bash\npython zetsu.py ingest\n```\n\n**4. Use it**\n\n```bash\npython zetsu.py tui # terminal UI\npython zetsu.py web # browser at localhost:8000\npython zetsu.py ask \"how do i abuse SeImpersonatePrivilege\"\n```\n\n---\n\n## CLI reference\n\n```bash\npython zetsu.py ingest # ingest enabled sources\npython zetsu.py ingest --force # wipe and rebuild everything\npython zetsu.py ingest --dry-run # estimate without making LLM calls\npython zetsu.py ingest --all-sources # enable all sources regardless of enabled flag\npython zetsu.py tui # terminal UI\npython zetsu.py web # web UI\npython zetsu.py ask \"query\" # one-shot CLI\npython zetsu.py ask \"query\" --style concept\npython zetsu.py stats # vector store stats\n```\n\n---\n\n## TUI keybindings\n\n| Key | Action |\n|-----|--------|\n| `Enter` | Send query |\n| `Ctrl+M` | Toggle Operator / Concept mode |\n| `Ctrl+Y` | Copy last response to clipboard |\n| `Ctrl+L` | Clear session |\n| `Ctrl+C` | Quit |\n| `/help` | Show commands |\n| `/stats` | Vector store + history stats |\n| `/history` | Today's past queries |\n| `/clear` | Clear session |\n\n---\n\n## Source types\n\n| Type | Use for | Extract mode |\n|------|---------|--------------|\n| `markdown_dir` | Your writeups, notes | `farr` |\n| `markdown_file` | Single markdown file | `farr` |\n| `url` | Tool wikis, reference pages | `headers` |\n| `atom` | Security blogs with feeds | `farr+narrative` |\n\n### Extract modes\n\n- **`farr`:** LLM extracts discrete attack steps as structured JSON (Finding, Action, Reasoning, Result). Best for writeups with clear attack chains.\n- **`farr+narrative`:** same as farr, plus a prose summary capturing the author's reasoning. Best for blogs where the thought process matters.\n- **`headers`:** no LLM, just split on headers and keep verbatim. Best for command references and tool documentation.\n\n---\n\n## LLM backends\n\nConfigure in `config.toml`. Separate backends for ingestion (bulk extraction) and queries (generation):\n\n```toml\n[llm] # query time\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\n\n[ingest] # ingest time\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\nworkers = 8 # parallel files during FARR extraction\n```\n\nSupported backends: `anthropic`, `openai` (any OpenAI-compatible API), `ollama`.\n\n---\n\n> [!IMPORTANT]\n> ChromaDB is run locally by default, which loads the full vector index into RAM. For large corpora (10k+ chunks) or memory-constrained machines, it is strongly recommended to run ChromaDB as a remote server instead. See [ChromaDB server docs](https://docs.trychroma.com/production/chroma-server/client-server-mode) for setup. Remote support in ZETSU is planned.\n\n> [!NOTE]\n> **Clipboard:** `Ctrl+Y` uses `wl-copy`. Install with `sudo pacman -S wl-clipboard` on Arch/Wayland.\n\n---\n\n## References\n\n- [rank-bm25](https://github.com/dorianbrown/rank_bm25), BM25 retrieval\n- FARR concept from [CIPHER](https://github.com/ibndias/CIPHER)\n- RAG pipeline inspired by [rag-chatbot](https://github.com/umbertogriffo/rag-chatbot)\n"}, "0482a882da1538854f40f50650783dd9dad31e08": {"t": 1785762671.2672615, "n": 6241, "h": "66ae77d4170944a9", "c": "

\n \"ZETSU\"\n

\n\nA personal offline RAG system for offensive security knowledge. Ingest your writeups, lab solutions, bug bounty reports, and security blogs, then query them in natural language during engagements. No cloud, no SaaS, your knowledge stays yours.\n\n\n---\n\n## What it does\n\nZETSU turns your accumulated knowledge into a queryable assistant. Instead of grepping through markdown files or remembering which writeup had that certipy command, you ask naturally:\n\n```\nhow do i escalate with SeImpersonatePrivilege\nwhat did i do after getting ADFS access\nsliver socks5 pivot setup\nexplain ESC8 vs ESC4\n```\n\nIt retrieves from your actual notes first, then generates an answer grounded in what you've documented, not generic internet knowledge.\n\n---\n\n## Architecture\n\n### Ingestion pipeline\n\n![Ingestion pipeline](assets/ingestion.png)\n\n### Query pipeline\n\n![Query pipeline](assets/query.png)\n\n---\n\n## Features\n\n- **FARR extraction:** at ingest time, an LLM reads each section of your writeups and extracts structured attack steps (Finding, Action, Reasoning, Result). What you retrieve is a semantic unit, not a random 800-token window.\n- **Hybrid retrieval:** BM25 for exact token matching (tool names, CVE numbers, CLI flags) + vector search for semantic similarity. RRF fusion combines both.\n- **Cross-encoder reranker:** reorders retrieved chunks by actual relevance before sending to the LLM.\n- **Two response styles:** Operator mode leads with exact commands, Concept mode leads with reasoning. Same retrieval, different presentation. Toggle with `Ctrl+M` in TUI.\n- **Multiple LLM backends:** Anthropic, OpenAI-compatible (DeepSeek, etc.), or local Ollama. Separate backends for ingest and query.\n- **Multiple source types:** local markdown files, single URLs, GitHub wikis, Atom/RSS feeds.\n- **Persistent chat history:** saved to `~/.zetsu/history/YYYY-MM-DD.json`, browsable with `/history`.\n- **TUI + Web UI + CLI:** use what fits the context. TUI during engagements, Web UI for studying, CLI for scripting.\n\n---\n\n## Installation\n\n```bash\ngit clone https://github.com/chaelsoo/zetsu\ncd zetsu\npip install -r requirements.txt\n```\n\nSet up your API key:\n\n```bash\ncp .env.example .env\n# edit .env and add your key\n```\n\n---\n\n## Quick start\n\n**1. Add your writeups**\n\nDrop `.md` files into `docs/`. Notion exports, Obsidian notes, blog exports, anything markdown works.\n\n**2. Configure sources in `config.toml`**\n\n```toml\n[llm]\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\n\n[[sources]]\ntype = \"markdown_dir\"\npath = \"./docs\"\nname = \"my_writeups\"\nextract = \"farr\"\nenabled = true\n```\n\n**3. Ingest**\n\n```bash\npython zetsu.py ingest\n```\n\n**4. Use it**\n\n```bash\npython zetsu.py tui # terminal UI\npython zetsu.py web # browser at localhost:8000\npython zetsu.py ask \"how do i abuse SeImpersonatePrivilege\"\n```\n\n---\n\n## CLI reference\n\n```bash\npython zetsu.py ingest # ingest enabled sources\npython zetsu.py ingest --force # wipe and rebuild everything\npython zetsu.py ingest --dry-run # estimate without making LLM calls\npython zetsu.py ingest --all-sources # enable all sources regardless of enabled flag\npython zetsu.py tui # terminal UI\npython zetsu.py web # web UI\npython zetsu.py ask \"query\" # one-shot CLI\npython zetsu.py ask \"query\" --style concept\npython zetsu.py stats # vector store stats\n```\n\n---\n\n## TUI keybindings\n\n| Key | Action |\n|-----|--------|\n| `Enter` | Send query |\n| `Ctrl+M` | Toggle Operator / Concept mode |\n| `Ctrl+Y` | Copy last response to clipboard |\n| `Ctrl+L` | Clear session |\n| `Ctrl+C` | Quit |\n| `/help` | Show commands |\n| `/stats` | Vector store + history stats |\n| `/history` | Today's past queries |\n| `/clear` | Clear session |\n\n---\n\n## Source types\n\n| Type | Use for | Extract mode |\n|------|---------|--------------|\n| `markdown_dir` | Your writeups, notes | `farr` |\n| `markdown_file` | Single markdown file | `farr` |\n| `url` | Tool wikis, reference pages | `headers` |\n| `atom` | Security blogs with feeds | `farr+narrative` |\n\n### Extract modes\n\n- **`farr`:** LLM extracts discrete attack steps as structured JSON (Finding, Action, Reasoning, Result). Best for writeups with clear attack chains.\n- **`farr+narrative`:** same as farr, plus a prose summary capturing the author's reasoning. Best for blogs where the thought process matters.\n- **`headers`:** no LLM, just split on headers and keep verbatim. Best for command references and tool documentation.\n\n---\n\n## LLM backends\n\nConfigure in `config.toml`. Separate backends for ingestion (bulk extraction) and queries (generation):\n\n```toml\n[llm] # query time\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\n\n[ingest] # ingest time\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\nworkers = 8 # parallel files during FARR extraction\n```\n\nSupported backends: `anthropic`, `openai` (any OpenAI-compatible API), `ollama`.\n\n---\n\n> [!IMPORTANT]\n> ChromaDB is run locally by default, which loads the full vector index into RAM. For large corpora (10k+ chunks) or memory-constrained machines, it is strongly recommended to run ChromaDB as a remote server instead. See [ChromaDB server docs](https://docs.trychroma.com/production/chroma-server/client-server-mode) for setup. Remote support in ZETSU is planned.\n\n> [!NOTE]\n> **Python 3.14:** a `tqdm` multiprocessing compatibility fix is already included in `zetsu.py`. No action needed.\n> **Clipboard:** `Ctrl+Y` uses `wl-copy`. Install with `sudo pacman -S wl-clipboard` on Arch/Wayland.\n\n---\n\n## Built with\n\n- [ChromaDB](https://github.com/chroma-core/chroma), vector store\n- [sentence-transformers](https://github.com/UKPLab/sentence-transformers), embeddings + reranker\n- [rank-bm25](https://github.com/dorianbrown/rank_bm25), BM25 retrieval\n- FARR concept from [CIPHER](https://github.com/ibndias/CIPHER)\n- RAG pipeline inspired by [rag-chatbot](https://github.com/umbertogriffo/rag-chatbot)\n"}}, "config.toml": {"de71caf42e56f9fec926fc6ca25195f055da09f0": {"t": 1785746798.5132613, "n": 6801, "h": "37d070e0ad5d8721", "c": "# ╔══════════════════════════════════════════════════════════════╗\n# ║ RAG — Personal Security Knowledge Assistant ║\n# ║ Edit this file to configure everything. No code changes. ║\n# ╚══════════════════════════════════════════════════════════════╝\n\n# ── Query LLM ────────────────────────────────────────────────────────────────\n# Used at query time to generate answers.\n# \"anthropic\" | \"ollama\" | \"openai\"\n[llm]\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\ntemperature = 0.3\nmax_tokens = 2000\n# openai_api_key = \"sk-...\" # or set OPENAI_API_KEY env var\n\n# ── Ingest LLM ───────────────────────────────────────────────────────────────\n# Used at ingest time for FARR extraction + narrative compression.\n[ingest]\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\nworkers = 8 # concurrent files processed in parallel during FARR extraction\n# api key falls back to OPENAI_API_KEY env var\n\n# ── Embeddings ───────────────────────────────────────────────────────────────\n[embeddings]\nmodel = \"all-MiniLM-L6-v2\" # fast, 22MB, good for security text\n# model = \"all-mpnet-base-v2\" # slower, better quality\nchunk_size = 800\nchunk_overlap = 100\n\n# ── Vector Store ─────────────────────────────────────────────────────────────\n[vectorstore]\npath = \"./chroma_db\"\ncollection = \"rag_main\"\n\n# ── Retrieval ────────────────────────────────────────────────────────────────\n[retrieval]\n# BM25 + vector candidates before RRF fusion\ncandidate_pool = 20\n# final chunks sent to LLM after reranking\ntop_k = 6\n# BM25 weight in RRF (0.0 = vector only, 1.0 = BM25 only)\nbm25_weight = 0.4\nvector_weight = 0.6\n# cross-encoder reranker — adds ~200ms but worth it\nreranker = true\nreranker_model = \"cross-encoder/ms-marco-MiniLM-L-2-v2\"\n\n# ── Response Styles ──────────────────────────────────────────────────────────\n# The toggle in TUI/Web UI switches between these.\n# Both retrieve the same chunks — only the presentation changes.\n[styles]\ndefault = \"operator\" # which style loads on startup\n\n[styles.operator]\nlabel = \"⚡ Operator\"\nprompt = \"\"\"You are a personal offensive security assistant for an expert penetration tester.\nThe user wants exact, actionable information.\n\nRules:\n- Lead with the exact command or technique. Put it in a code block immediately.\n- Explain AFTER the command, not before.\n- Be terse. No preamble, no disclaimers.\n- If multiple commands are needed, show the full sequence in order.\n- When context includes FARR-structured steps (Finding/Action/Reasoning/Result), extract the Action first.\n- If the exact command isn't in the retrieved context, say so clearly before falling back to general knowledge.\n\nIf the user's message is a greeting, casual remark, or unrelated to offensive security,\nrespond conversationally and do not reference the retrieved context.\n\"\"\"\n\n[styles.concept]\nlabel = \"📖 Concept\"\nprompt = \"\"\"You are a personal offensive security assistant for an expert penetration tester.\nThe user wants to understand how something works.\n\nRules:\n- Lead with the core concept and why it matters.\n- Use retrieved writeup context as ground truth — prefer the author's reasoning over general knowledge.\n- Walk through the attack chain or technical mechanism step by step.\n- Use commands as illustrations, not the focus.\n- Cover: what it is, why it works, when to use it, what can go wrong.\n- Keep it dense — no filler, no basic definitions the user already knows.\n\nIf the user's message is a greeting, casual remark, or unrelated to offensive security,\nrespond conversationally and do not reference the retrieved context.\n\"\"\"\n\n# ── Web UI ───────────────────────────────────────────────────────────────────\n[webui]\nhost = \"127.0.0.1\"\nport = 8000\n\n# ── Sources ──────────────────────────────────────────────────────────────────\n# extract modes:\n# \"farr\" — LLM extracts attack steps as FARR JSON (writeups, CTF notes)\n# \"farr+narrative\"— FARR extraction + narrative compression (blogs, 0xdf-style)\n# \"headers\" — universal header split, no LLM (tool wikis, reference docs)\n\n[[sources]]\ntype = \"markdown_dir\"\npath = \"./docs\"\nname = \"local_notes\"\nextract = \"farr\"\nenabled = true\n\n[[sources]]\ntype = \"url\"\nurl = \"https://github.com/BishopFox/sliver/wiki\"\nname = \"sliver_wiki\"\nextract = \"headers\"\nenabled = true\n\n[[sources]]\ntype = \"url\"\nurl = \"https://book.hacktricks.wiki/windows-hardening/active-directory-methodology/ad-certificates/domain-escalation\"\nname = \"hacktricks_adcs\"\nextract = \"farr+narrative\"\nenabled = true\n\n# ── Adding more sources ───────────────────────────────────────────────────────\n# To add your own writeups, drop .md files into ./docs/ and re-run ingest.\n\n# ── Blogs (Atom/RSS feeds) ────────────────────────────────────────────────────\n\n# 0xdf — HTB writeups\n# [[sources]]\n# type = \"atom\"\n# url = \"https://0xdf.gitlab.io/feed.xml\"\n# name = \"0xdf\"\n# extract = \"farr+narrative\"\n# limit = 50\n# enabled = false\n\n# dirkjanm — original AD/Azure research (RBCD, shadow credentials, AADInternals)\n[[sources]]\ntype = \"atom\"\nurl = \"https://dirkjanm.io/feed.xml\"\nname = \"dirkjanm\"\nextract = \"farr+narrative\"\nlimit = 30\nenabled = false\n\n# harmj0y — Kerberos, delegation, BloodHound original research\n[[sources]]\ntype = \"atom\"\nurl = \"https://blog.harmj0y.net/feed/\"\nname = \"harmj0y\"\nextract = \"farr+narrative\"\nlimit = 30\nenabled = false\n\n# Dominic Breuker — Sliver C2 deep dives\n[[sources]]\ntype = \"atom\"\nurl = \"https://dominicbreuker.com/index.xml\"\nname = \"dominicbreuker\"\nextract = \"farr+narrative\"\nlimit = 30\nenabled = false\n\n# Shenanigans Labs — original AD/Azure research\n[[sources]]\ntype = \"atom\"\nurl = \"https://shenaniganslabs.io/feed.xml\"\nname = \"shenaniganslabs\"\nextract = \"farr+narrative\"\nlimit = 30\nenabled = false\n\n# Black Hills Infosec — mixed security blog\n[[sources]]\ntype = \"atom\"\nurl = \"https://www.blackhillsinfosec.com/feed/\"\nname = \"bhis\"\nextract = \"farr+narrative\"\nlimit = 20\nenabled = false\n\n# ── Single pages / reference material ────────────────────────────────────────\n\n# CPTS tips — single page\n[[sources]]\ntype = \"url\"\nurl = \"https://www.brunorochamoura.com/posts/cpts-tips/\"\nname = \"cpts-tips\"\nextract = \"farr\"\nenabled = false\n\n# NetExec for AD pentesting — single page cheatsheet\n[[sources]]\ntype = \"url\"\nurl = \"https://www.hackingarticles.in/netexec-for-oscp-ad-pentesting/\"\nname = \"netexec-cheatsheet\"\nextract = \"headers\"\nenabled = false\n\n# bloodyAD wiki — tool reference\n[[sources]]\ntype = \"url\"\nurl = \"https://github.com/CravateRouge/bloodyAD/wiki/User-Guide\"\nname = \"bloodyad-wiki\"\nextract = \"headers\"\nenabled = false\n"}, "0482a882da1538854f40f50650783dd9dad31e08": {"t": 1785762661.9545193, "n": 6485, "h": "78cd9a50b96e58a8", "c": "# ╔══════════════════════════════════════════════════════════════╗\n# ║ RAG — Personal Security Knowledge Assistant ║\n# ║ Edit this file to configure everything. No code changes. ║\n# ╚══════════════════════════════════════════════════════════════╝\n\n# ── Query LLM ────────────────────────────────────────────────────────────────\n# Used at query time to generate answers.\n# \"anthropic\" | \"ollama\" | \"openai\"\n[llm]\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\ntemperature = 0.3\nmax_tokens = 2000\n# openai_api_key = \"sk-...\" # or set OPENAI_API_KEY env var\n\n# ── Ingest LLM ───────────────────────────────────────────────────────────────\n# Used at ingest time for FARR extraction + narrative compression.\n[ingest]\nbackend = \"openai\"\nopenai_model = \"deepseek-chat\"\nbase_url = \"https://api.deepseek.com/v1\"\nworkers = 8 # concurrent files processed in parallel during FARR extraction\n# api key falls back to OPENAI_API_KEY env var\n\n# ── Embeddings ───────────────────────────────────────────────────────────────\n[embeddings]\nmodel = \"all-MiniLM-L6-v2\" # fast, 22MB, good for security text\n# model = \"all-mpnet-base-v2\" # slower, better quality\nchunk_size = 800\nchunk_overlap = 100\n\n# ── Vector Store ─────────────────────────────────────────────────────────────\n[vectorstore]\npath = \"./chroma_db\"\ncollection = \"rag_main\"\n\n# ── Retrieval ────────────────────────────────────────────────────────────────\n[retrieval]\n# BM25 + vector candidates before RRF fusion\ncandidate_pool = 20\n# final chunks sent to LLM after reranking\ntop_k = 6\n# BM25 weight in RRF (0.0 = vector only, 1.0 = BM25 only)\nbm25_weight = 0.4\nvector_weight = 0.6\n# cross-encoder reranker — adds ~200ms but worth it\nreranker = true\nreranker_model = \"cross-encoder/ms-marco-MiniLM-L-2-v2\"\n\n# ── Response Styles ──────────────────────────────────────────────────────────\n# The toggle in TUI/Web UI switches between these.\n# Both retrieve the same chunks — only the presentation changes.\n[styles]\ndefault = \"operator\" # which style loads on startup\n\n[styles.operator]\nlabel = \"⚡ Operator\"\nprompt = \"\"\"You are a personal offensive security assistant for an expert penetration tester.\nThe user wants exact, actionable information.\n\nRules:\n- Lead with the exact command or technique. Put it in a code block immediately.\n- Explain AFTER the command, not before.\n- Be terse. No preamble, no disclaimers.\n- If multiple commands are needed, show the full sequence in order.\n- When context includes FARR-structured steps (Finding/Action/Reasoning/Result), extract the Action first.\n- If the exact command isn't in the retrieved context, say so clearly before falling back to general knowledge.\n\"\"\"\n\n[styles.concept]\nlabel = \"📖 Concept\"\nprompt = \"\"\"You are a personal offensive security assistant for an expert penetration tester.\nThe user wants to understand how something works.\n\nRules:\n- Lead with the core concept and why it matters.\n- Use retrieved writeup context as ground truth — prefer the author's reasoning over general knowledge.\n- Walk through the attack chain or technical mechanism step by step.\n- Use commands as illustrations, not the focus.\n- Cover: what it is, why it works, when to use it, what can go wrong.\n- Keep it dense — no filler, no basic definitions the user already knows.\n\"\"\"\n\n# ── Web UI ───────────────────────────────────────────────────────────────────\n[webui]\nhost = \"127.0.0.1\"\nport = 8000\n\n# ── Sources ──────────────────────────────────────────────────────────────────\n# extract modes:\n# \"farr\" — LLM extracts attack steps as FARR JSON (writeups, CTF notes)\n# \"farr+narrative\"— FARR extraction + narrative compression (blogs, 0xdf-style)\n# \"headers\" — universal header split, no LLM (tool wikis, reference docs)\n\n[[sources]]\ntype = \"markdown_dir\"\npath = \"./docs\"\nname = \"local_notes\"\nextract = \"farr\"\nenabled = true\n\n[[sources]]\ntype = \"url\"\nurl = \"https://github.com/BishopFox/sliver/wiki\"\nname = \"sliver_wiki\"\nextract = \"headers\"\nenabled = true\n\n[[sources]]\ntype = \"url\"\nurl = \"https://book.hacktricks.wiki/windows-hardening/active-directory-methodology/ad-certificates/domain-escalation\"\nname = \"hacktricks_adcs\"\nextract = \"farr+narrative\"\nenabled = true\n\n# ── Adding more sources ───────────────────────────────────────────────────────\n# To add your own writeups, drop .md files into ./docs/ and re-run ingest.\n\n# ── Blogs (Atom/RSS feeds) ────────────────────────────────────────────────────\n\n# 0xdf — HTB writeups\n# [[sources]]\n# type = \"atom\"\n# url = \"https://0xdf.gitlab.io/feed.xml\"\n# name = \"0xdf\"\n# extract = \"farr+narrative\"\n# limit = 50\n# enabled = false\n\n# dirkjanm — original AD/Azure research (RBCD, shadow credentials, AADInternals)\n[[sources]]\ntype = \"atom\"\nurl = \"https://dirkjanm.io/feed.xml\"\nname = \"dirkjanm\"\nextract = \"farr+narrative\"\nlimit = 30\nenabled = false\n\n# harmj0y — Kerberos, delegation, BloodHound original research\n[[sources]]\ntype = \"atom\"\nurl = \"https://blog.harmj0y.net/feed/\"\nname = \"harmj0y\"\nextract = \"farr+narrative\"\nlimit = 30\nenabled = false\n\n# Dominic Breuker — Sliver C2 deep dives\n[[sources]]\ntype = \"atom\"\nurl = \"https://dominicbreuker.com/index.xml\"\nname = \"dominicbreuker\"\nextract = \"farr+narrative\"\nlimit = 30\nenabled = false\n\n# Shenanigans Labs — original AD/Azure research\n[[sources]]\ntype = \"atom\"\nurl = \"https://shenaniganslabs.io/feed.xml\"\nname = \"shenaniganslabs\"\nextract = \"farr+narrative\"\nlimit = 30\nenabled = false\n\n# Black Hills Infosec — mixed security blog\n[[sources]]\ntype = \"atom\"\nurl = \"https://www.blackhillsinfosec.com/feed/\"\nname = \"bhis\"\nextract = \"farr+narrative\"\nlimit = 20\nenabled = false\n\n# ── Single pages / reference material ────────────────────────────────────────\n\n# CPTS tips — single page\n[[sources]]\ntype = \"url\"\nurl = \"https://www.brunorochamoura.com/posts/cpts-tips/\"\nname = \"cpts-tips\"\nextract = \"farr\"\nenabled = false\n\n# NetExec for AD pentesting — single page cheatsheet\n[[sources]]\ntype = \"url\"\nurl = \"https://www.hackingarticles.in/netexec-for-oscp-ad-pentesting/\"\nname = \"netexec-cheatsheet\"\nextract = \"headers\"\nenabled = false\n\n# bloodyAD wiki — tool reference\n[[sources]]\ntype = \"url\"\nurl = \"https://github.com/CravateRouge/bloodyAD/wiki/User-Guide\"\nname = \"bloodyad-wiki\"\nextract = \"headers\"\nenabled = false\n"}}}}