1 line
15 KiB
JSON
1 line
15 KiB
JSON
{"repo": "2dogsandanerd/clawrag", "files": {"README.md": {"2de62983c5f8097c5cab15fbb59be6e283bd47fd": {"t": 1785596407.7609417, "n": 13394, "h": "6ac94e1769b00893", "c": "⚠️ IMPORTANT NOTICE REGARDING PLAGIARISM & AUTHORSHIP ⚠️ \n\n\nIt has come to my attention that the GitHub user @tfantas (Thiago Antas) and his automated account @jarvis-aix are falsely claiming credit for my architecture.\nThey have explicitly listed my original repositories (including RAG_enterprise_core, smart-ingest-kit, DAUT, etc.) as their own '🔬 Featured Work' on their public \nprofile without authorization or proper attribution. Below is the documented proof.\n\n\n\n\nhttps://github.com/tfantas seems to have 20+ years of expirience but no own ideas .... Im gonna make him famous...... \nIf you enjoyed my repos and found them useful, Im sorry but im out of this game !!! No more opensource Sorry\nIm sure you will find my further developed Repos at https://github.com/jarvis-aix .... What a disgrace and disrespect !\n\nThis repository, the Multi-Lane Consensus Architecture, and the V4.0 Manifest are 100% my original work, built over two years. \nPlease be highly cautious of actors in the AI space attempting to rebrand, clone, or take credit for this Enterprise RAG system\n\n⚠️ ⚠️ ⚠️ \n\n\n\n\n\n\n# ClawRAG\n\n[](https://github.com/2dogsandanerd/ClawRag/releases)\n[](https://opensource.org/licenses/MIT)\n[](https://www.docker.com/)\n[](https://www.python.org/)\n[](https://github.com/2dogsandanerd/RAG_enterprise_core)\n\n**Production-Ready RAG Engine. Self-Hosted. Data Sovereign. Enterprise-Grade.**\n\nClawRAG is a powerful, self-hosted Retrieval-Augmented Generation (RAG) system that gives you complete control over your AI document processing. Process documents locally with local LLMs or connect to cloud APIs—your data never leaves your infrastructure unless you want it to.\n\n[🚀 Quick Start](#quick-start) • [📊 Editions](#editions) • [🏢 Enterprise](#enterprise-features) • [📖 Documentation](docs/)\n\n---\n\n## 🎯 Why ClawRAG?\n\n### The Problem\nEnterprise document processing forces you to choose:\n- ❌ **Cloud solutions**: Send sensitive data to third parties\n- ❌ **Simple tools**: Fail on complex PDFs, tables, mixed content\n- ❌ **DIY approaches**: Months of integration, no production readiness\n\n### Our Solution\nClawRAG provides **production-grade document intelligence** that runs entirely on your infrastructure:\n\n✅ **Self-Hosted**: Complete data sovereignty \n✅ **Intelligent Processing**: Handles complex PDFs, tables, scanned documents \n✅ **Hybrid Search**: Combines semantic + keyword search for accuracy \n✅ **Production-Ready**: Docker-first, scalable, monitored \n✅ **Enterprise Path**: Seamless upgrade to advanced features \n\n---\n\n## 🚀 Quick Start\n\n### Prerequisites\n- Docker & Docker Compose installed\n- 8GB+ RAM (16GB recommended)\n\n### One-Command Setup\n\n```bash\n# 1. Clone and enter\ngit clone https://github.com/2dogsandanerd/ClawRag.git\ncd ClawRag\n\n# 2. Configure\ncp .env.example .env\n\n# 3. Start everything\ndocker compose up -d\n\n# 4. Verify\ncurl http://localhost:8080/health\n```\n\n**Access your instance:**\n- 🌐 **Web UI**: http://localhost:8080\n- 📚 **API Docs**: http://localhost:8080/docs\n- 🔍 **Health Check**: http://localhost:8080/health\n\n---\n\n## 📊 Editions\n\nChoose the edition that fits your needs:\n\n| Feature | Community Edition | Enterprise Edition |\n|---------|------------------|-------------------|\n| **Document Processing** | Single-engine (Docling) | Multi-engine consensus |\n| **Supported Formats** | PDF, DOCX, MD, TXT, CSV | + Images, Email, Code, PPT, XLS |\n| **Search** | Vector + BM25 Hybrid | + Graph RAG, Semantic Routing |\n| **Data Validation** | Basic quality scoring | Consensus validation, 100% accuracy |\n| **Human Verification** | Manual review | Surgical precision (only conflicts) |\n| **Multi-Tenancy** | Single user | Mission-based isolation |\n| **Quality Assurance** | Basic logging | Real-time dashboards, automated testing |\n| **Support** | Community | SLA-backed, dedicated support |\n| **Pricing** | Free (MIT License) | Custom enterprise licensing |\n\n**[→ Compare full feature matrix](docs/editions.md)**\n\n---\n\n## 🏢 Enterprise Features\n\nNeed more? [ClawRAG Enterprise Core](https://github.com/2dogsandanerd/RAG_enterprise_core) provides advanced capabilities:\n\n### 🧠 Adaptive Intelligence\n- **Multi-Engine Processing**: 3+ specialized extractors work in parallel\n- **Consensus Validation**: Automated comparison ensures 100% data integrity\n- **Intelligent Routing**: Analyzes document type and selects optimal strategy\n\n### 🔒 Zero Data Loss Architecture\n- **Parallel Processing**: Multiple engines analyze each document independently\n- **Conflict Detection**: Flags discrepancies for targeted human review\n- **Visual Verification**: See exactly where conflicts occur on source documents\n\n### 🌐 Advanced Knowledge Retrieval\n- **Graph RAG**: Neo4j-powered relationship traversal between documents\n- **Semantic + Graph Hybrid**: Combines concept search with factual relationships\n- **Query Decomposition**: Complex queries automatically split into sub-tasks\n\n### 📊 Mission-Based Multi-Tenancy\n- **Customer Isolation**: Complete data separation per mission\n- **Hot Configuration**: Update rules without system restart\n- **Quality Gates**: Per-mission thresholds and processing rules\n\n### 📈 Continuous Quality Assurance\n- **Real-Time Observability**: Grafana dashboards for all processing stages\n- **Automated Testing**: Continuous validation against reference datasets\n- **Performance Monitoring**: Track accuracy, speed, and system health\n\n**[→ View Enterprise Manifest](https://github.com/2dogsandanerd/RAG_enterprise_core/blob/main/manifest_v4.0.md)**\n\n---\n\n## 💼 Use Cases\n\n### 📄 Legal & Compliance\nProcess contracts, court filings, and regulatory documents with:\n- Citation extraction and validation\n- Clause detection and comparison\n- Audit-compliant processing trails\n- PII detection and redaction\n\n### 🏥 Healthcare & Research\nAnalyze medical literature and patient records:\n- Medical entity extraction\n- Cross-document concept linking\n- Citation graph analysis\n- HIPAA-compliant processing\n\n### 💰 Financial Services\nProcess invoices, reports, and due diligence documents:\n- Entity mapping and verification\n- Anomaly detection in financial data\n- Structured data extraction from tables\n- Multi-document reconciliation\n\n### 🔬 Technical Documentation\nIngest API docs, manuals, and technical specifications:\n- Code block preservation\n- Structured content extraction\n- Semantic chunking for technical terms\n- Cross-reference resolution\n\n---\n\n## 🏗️ Architecture\n\n```\n┌─────────────────────────────────────────────────────────┐\n│ Nginx Gateway (8080) │\n│ ┌──────────────┐ ┌──────────────────────────────┐ │\n│ │ Frontend UI │ │ FastAPI Backend (8081) │ │\n│ │ (Vue.js) │ │ - RAG API │ │\n│ └──────────────┘ │ - Document Processing │ │\n│ │ - Hybrid Search │ │\n│ └──────────────┬───────────────┘ │\n└───────────────────────────────────┼─────────────────────┘\n │\n ┌──────────────────────────┼──────────────────┐\n │ │ │\n ▼ ▼ ▼\n┌─────────────────┐ ┌──────────────────────┐ ┌──────────────┐\n│ ChromaDB │ │ Ollama / LLM │ │ (Optional) │\n│ Vector Store │ │ Embeddings & Chat │ │ Enterprise │\n│ (Port 8001) │ │ (Port 11434) │ │ Extensions │\n└─────────────────┘ └──────────────────────┘ └──────────────┘\n```\n\n### Key Components\n\n**Backend (`backend/src/`):**\n- `api/v1/rag/` - REST API endpoints (ingestion, query, collections)\n- `core/` - ChromaDB manager, document loaders, retrievers\n- `services/` - Document processing, deduplication, task management\n\n**Processing Pipeline:**\n1. **Ingestion**: Multi-format support with intelligent parsing\n2. **Chunking**: Configurable semantic and sentence-based strategies\n3. **Embedding**: Multi-provider support (Ollama, OpenAI, Anthropic)\n4. **Storage**: ChromaDB with metadata and hybrid search\n5. **Retrieval**: Vector + BM25 with Reciprocal Rank Fusion\n\n---\n\n## ⚙️ Configuration\n\n### Environment Variables\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `PORT` | `8080` | External port for nginx gateway |\n| `DOCS_DIR` | `./data/docs` | Host directory for folder ingestion |\n| `LLM_PROVIDER` | `ollama` | LLM provider (ollama, openai, anthropic, gemini) |\n| `LLM_MODEL` | `llama3:latest` | Model name for selected provider |\n| `EMBEDDING_MODEL` | `nomic-embed-text` | Embedding model name |\n| `CHUNK_SIZE` | `512` | Size of text chunks |\n| `CHUNK_OVERLAP` | `128` | Overlap between chunks |\n| `DEBUG` | `false` | Enable debug logging |\n\n### LLM Configuration Examples\n\n**Local (Ollama) - Privacy First:**\n```bash\nLLM_PROVIDER=ollama\nLLM_MODEL=llama3:latest\nEMBEDDING_MODEL=nomic-embed-text\n```\n\n**Cloud (OpenAI) - Maximum Performance:**\n```bash\nLLM_PROVIDER=openai\nLLM_MODEL=gpt-4o\nOPENAI_API_KEY=sk-proj-...\n```\n\n**Local Server (LM Studio, etc.):**\n```bash\nLLM_PROVIDER=openai_compatible\nLLM_MODEL=local-model\nOPENAI_BASE_URL=http://host.docker.internal:1234/v1\n```\n\n---\n\n## 🛠️ Using the API\n\n### Python Example\n\n```python\nimport requests\n\nBASE_URL = \"http://localhost:8080/api/v1/rag\"\n\n# 1. Create a collection\nrequests.post(f\"{BASE_URL}/collections\", files={\n \"collection_name\": (None, \"legal_docs\"),\n \"embedding_model\": (None, \"nomic-embed-text\")\n})\n\n# 2. Upload documents\nwith open(\"contract.pdf\", \"rb\") as f:\n requests.post(\n f\"{BASE_URL}/documents/upload\",\n files={\"files\": f},\n data={\"collection_name\": \"legal_docs\"}\n )\n\n# 3. Query knowledge base\nresponse = requests.post(\n f\"{BASE_URL}/query\",\n json={\n \"query\": \"What are the termination clauses?\",\n \"collection\": \"legal_docs\",\n \"k\": 5\n }\n)\nprint(response.json()[\"answer\"])\n```\n\n### cURL Examples\n\n```bash\n# Health check\ncurl http://localhost:8080/health\n\n# List collections\ncurl http://localhost:8080/api/v1/rag/collections\n\n# Query with filters\ncurl -X POST http://localhost:8080/api/v1/rag/query \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"query\": \"Explain the architecture\",\n \"collection\": \"my_docs\",\n \"k\": 10,\n \"similarity_threshold\": 0.5\n }'\n```\n\n---\n\n## 📦 What's Included\n\n### Community Edition\n- ✅ FastAPI backend with comprehensive API\n- ✅ Web UI for document management and querying\n- ✅ ChromaDB vector database\n- ✅ Ollama LLM/embedding service\n- ✅ Nginx reverse proxy\n- ✅ Multi-provider LLM support\n- ✅ Hybrid search (vector + BM25)\n- ✅ Document chunking strategies\n- ✅ Folder batch ingestion\n\n### Optional Enterprise Extensions\n- 🔧 Multi-engine consensus processing\n- 🔧 Graph database (Neo4j) integration\n- 🔧 Real-time monitoring dashboards\n- 🔧 Advanced quality assurance pipeline\n- 🔧 Mission-based multi-tenancy\n- 🔧 PII detection and compliance tools\n\n---\n\n## 🚨 Troubleshooting\n\n### Common Issues\n\n**LLM Connection Problems:**\n```bash\n# Check LLM initialization\ndocker compose logs backend | grep \"LLM\"\n\n# For local servers, verify network\ndocker exec clawrag-backend curl http://host.docker.internal:11434\n```\n\n**Folder Ingestion Not Working:**\n```bash\n# Verify DOCS_DIR is set correctly\ncat .env | grep DOCS_DIR\n\n# Check mount inside container\ndocker exec clawrag-backend ls -la /host_root/\n```\n\n**Performance Issues:**\n```bash\n# Check resource usage\ndocker stats\n\n# Review logs for bottlenecks\ndocker compose logs -f backend\n```\n\n**[→ Full troubleshooting guide](docs/troubleshooting.md)**\n\n---\n\n## 🤝 Community & Support\n\n### Community Edition\n- 🐛 [GitHub Issues](https://github.com/2dogsandanerd/ClawRag/issues) - Bug reports\n- 💡 [Discussions](https://github.com/2dogsandanerd/ClawRag/discussions) - Feature requests\n- 📖 [Documentation](docs/) - Setup and configuration\n- 📝 [Contributing Guide](CONTRIBUTING.md) - How to contribute\n\n### Enterprise Support\n- 🎫 SLA-backed support with response time guarantees\n- 📞 Direct engineering contact\n- 🚀 Priority feature development\n- 🏢 Custom integration assistance\n\n**[→ Contact for Enterprise](mailto:2dogsandanerd@gmail.com)**\n\n---\n\n## 📄 License\n\n**Community Edition**: MIT License - Free for commercial and personal use.\n\n**Enterprise Edition**: Proprietary license with custom terms. Contact for details.\n\nCopyright (c) 2025 2dogsandanerd\n\n---\n\n## 🎯 Roadmap\n\n### Version 1.3 (Next)\n- [ ] Multi-collection search\n- [ ] Enhanced UI with query history\n- [ ] Additional document format support\n\n### Version 2.0 (Planned)\n- [ ] Conversational memory\n- [ ] Advanced analytics dashboard\n- [ ] Plugin system for custom processors\n\n### Enterprise Roadmap\n- [ ] Kubernetes deployment\n- [ ] Advanced graph reasoning\n- [ ] Multi-modal search (text + images)\n- [ ] ISO 27001 certification\n\n---\n\n<p align=\"center\">\n <strong>Built with ❤️ for developers who value data sovereignty</strong><br>\n <a href=\"https://github.com/2dogsandanerd/ClawRag\">GitHub</a> •\n <a href=\"https://github.com/2dogsandanerd/RAG_enterprise_core\">Enterprise</a> •\n <a href=\"docs/\">Documentation</a>\n</p>\n"}}}} |