{"repo": "algiras/rusty-pageindex", "files": {"README.md": {"f2bbdaab2903f7afb3a32af1e80e2c466c96bab1": {"t": 1785674477.6951747, "n": 6847, "h": "743042a3b25d863c", "c": "# 🦀 RustyPageIndex\n\n![Rusty Page Indexer Cover](assets/cover.png)\n\n**RustyPageIndex** is a high-performance Rust implementation of the PageIndex pattern. It transforms complex documents into hierarchical \"Table-of-Contents\" (TOC) trees for vectorless, reasoning-based RAG.\n\nThis project is inspired by [VectifyAI/PageIndex](https://github.com/VectifyAI/PageIndex) but has diverged significantly with multi-repo support, parallel processing, and a unified tree architecture.\n\n## 🚀 Key Features\n\n### Performance\n- **Parallel Indexing**: Uses Rayon for parallel file parsing (238 files in ~0.04s)\n- **Rust-Native Parsing**: `pdf-extract` and `pulldown-cmark` for fast document processing\n- **Incremental Updates**: Hash-based caching skips unchanged files\n\n### Multi-Repository Support\n- **Index multiple repos**: Each indexed folder is tracked separately\n- **Query across all**: Search spans all indexed repositories by default\n- **Manage indices**: List, filter, and clean up indices easily\n\n### Unified Tree Architecture\n- **Folder → File → Section** hierarchy preserves document structure\n- **Single tree per repo**: Efficient storage and navigation\n- **Smart search**: Auto-unwraps folder roots for better LLM context\n\n---\n\n## 🔄 Divergence from Original PageIndex\n\n| Feature | Original PageIndex | RustyPageIndex |\n|---------|-------------------|----------------|\n| Language | Python | Rust |\n| Indexing | Per-file indices | Unified folder tree |\n| Multi-repo | Not supported | Full support with `list`/`clean` |\n| Parallelism | Sequential | Rayon parallel processing |\n| Storage | Cloud-based (MCP) | Local filesystem |\n| Tree Structure | Flat sections | Folder → File → Section hierarchy |\n| Headerless Markdown | Empty tree | Auto-creates \"Document\" node |\n\n---\n\n## 🛠️ Getting Started\n\n### Installation\n\n**One-liner Install (Unix/macOS):**\n```bash\ncurl -fsSL https://raw.githubusercontent.com/Algiras/rusty-pageindex/main/install.sh | bash\n```\n\n**One-liner Install (Windows PowerShell):**\n```powershell\nirm https://raw.githubusercontent.com/Algiras/rusty-pageindex/main/install.ps1 | iex\n```\n\n**Via Cargo:**\n```bash\ncargo install rusty-page-indexer\n```\n\n### 🧙 Use as an Agent Skill\n```bash\nnpx skills add https://github.com/Algiras/rusty-pageindex --skill rusty-page-indexer\n```\n\n### 🔑 Authentication\n\n```bash\n# For OpenAI\nrusty-page-indexer auth --api-key \"your-key-here\"\n\n# For Ollama (local LLM)\nrusty-page-indexer auth --api-key \"ollama\" --api-base \"http://localhost:11434/v1\" --model \"llama3.2\"\n```\n\n---\n\n## 🌲 Usage\n\n### Indexing Documents\n```bash\n# Index a repository\nrusty-page-indexer index ./my-project\n\n# Index with LLM-generated summaries\nrusty-page-indexer index ./my-project --enrich\n\n# Force re-index (ignores cache)\nrusty-page-indexer index ./my-project --force\n\n# Preview what would be indexed\nrusty-page-indexer index ./my-project --dry-run\n```\n\n### Managing Multiple Repositories\n```bash\n# Index multiple repos\nrusty-page-indexer index ./repo-a\nrusty-page-indexer index ./repo-b\n\n# List all indexed repositories\nrusty-page-indexer list\n\n# Example output:\n# 📋 Indexed Repositories\n# ────────────────────────────────────────────────────────────\n#   📁 repo-a (125.3 KB)\n#      /Users/you/projects/repo-a\n#   📁 repo-b (89.7 KB)\n#      /Users/you/projects/repo-b\n# ────────────────────────────────────────────────────────────\n# Total: 2 indices\n```\n\n### Querying\n```bash\n# Search across ALL indexed repositories\nrusty-page-indexer query \"how does authentication work\"\n\n# Search within a specific repository\nrusty-page-indexer query \"kafka messaging\" --path repo-a\n```\n\n### Cleanup\n```bash\n# Remove a specific index\nrusty-page-indexer clean repo-a\n\n# Remove all indices\nrusty-page-indexer clean --all\n```\n\n### Status Information\n```bash\nrusty-page-indexer info\n```\n\n---\n\n## 🤖 Model Compatibility\n\n### OpenAI Models (Remote)\n\n| Model | Cost | Speed | Notes |\n|-------|------|-------|-------|\n| `gpt-4o` | $$$ | Fast | Best accuracy, recommended for complex queries |\n| `gpt-4o-mini` | $ | Very Fast | Great balance of cost and quality ⭐ |\n| `gpt-4.1-mini` | $ | Very Fast | Latest mini variant |\n| `gpt-4-turbo` | $$ | Fast | Good for detailed reasoning |\n| `gpt-3.5-turbo` | ¢ | Very Fast | Budget option, decent accuracy |\n\n```bash\n# Configure for OpenAI\nrusty-page-indexer auth --api-key \"sk-...\" --model \"gpt-4o-mini\"\n\n# Override model per query\nrusty-page-indexer query \"question\" --model gpt-4o\n```\n\n### Local Models (Ollama)\n\n| Model | Size | Works | Notes |\n|-------|------|-------|-------|\n| `gemma3:1b` | 1B | ✅ | Minimum recommended for local use |\n| `llama3.2:latest` | 3B | ✅ | Good balance of speed and accuracy ⭐ |\n| `qwen2.5:7b` | 7B | ✅ | Reliable, slightly conservative |\n| `llama3.1:latest` | 8B | ✅ | Excellent accuracy |\n| `mistral:7b` | 7B | ✅ | Fast and capable |\n| `phi3:mini` | 3.8B | ✅ | Microsoft's compact model |\n| `qwen2.5:0.5b` | 0.5B | ❌ | Too small, unreliable responses |\n| `tinyllama:1.1b` | 1.1B | ❌ | Doesn't follow output format |\n\n```bash\n# Configure for Ollama\nrusty-page-indexer auth --api-key \"ollama\" --api-base \"http://localhost:11434/v1\" --model \"llama3.2\"\n\n# Make sure Ollama is running\nollama serve\n```\n\n### OpenAI-Compatible APIs\n\nWorks with any OpenAI-compatible endpoint:\n\n```bash\n# Azure OpenAI\nrusty-page-indexer auth --api-key \"your-key\" --api-base \"https://your-resource.openai.azure.com/v1\" --model \"gpt-4\"\n\n# Together AI\nrusty-page-indexer auth --api-key \"your-key\" --api-base \"https://api.together.xyz/v1\" --model \"meta-llama/Llama-3-70b-chat-hf\"\n\n# Groq\nrusty-page-indexer auth --api-key \"your-key\" --api-base \"https://api.groq.com/openai/v1\" --model \"llama3-70b-8192\"\n```\n\n**Recommendation**: Use `gpt-4o-mini` for remote or `llama3.2` for local. Add `--enrich` during indexing for better search quality.\n\n---\n\n## 🔍 How Search Works\n\n1. **Repository Selection**: Query matches against all indexed repos (or filtered by `--path`)\n2. **Tree Navigation**: LLM navigates the Folder → File → Section hierarchy\n3. **Content Retrieval**: Matching leaf nodes return full text content\n\nThe unified tree structure allows the LLM to see file names within folders, making navigation more accurate than flat file lists.\n\n---\n\n## 📁 Storage Structure\n\n```\n~/.rusty-page-indexer/\n├── config.toml           # API credentials and settings\n├── manifest.json         # Index registry with all repos\n└── indices/\n    ├── {hash-a}.json     # Unified tree for repo-a\n    └── {hash-b}.json     # Unified tree for repo-b\n```\n\n---\n\n## 📁 Supported Document Types\n\n### Markdown (`.md`)\n- Parses heading structure (`#`, `##`, `###`) into hierarchical tree\n- Headerless files auto-create a \"Document\" node with full content\n\n### PDF (`.pdf`)\n- Extracts text using `pdf-extract`\n- Creates single root node with document text\n- Works best with text-based PDFs (not scanned images)\n\n---\n\n## 📄 License\nMIT\n"}}}}