{"repo": "angad2005/chatrag", "files": {"README.md": {"724b633d339313606349c80991147318edf104e1": {"t": 1785674476.047299, "n": 5907, "h": "fcf6f166061564b4", "c": "# 🚀 ChatRAG (Chatbot AI Agent & Knowledge Base Injector)\n\n> LLM-powered tool for working with your data files locally on your own system or server. Makes your LLM more knowledgeable using document injection.\n\n## 📌 Description\n\nThis project uses Python for both front-end (Streamlit) and back-end, connecting to any OpenAI-compatible LLM endpoint (LM Studio, vLLM, Ollama, NVIDIA NIM, OpenAI, etc.).\n\n- Chat with your personal private files using LLMs for fast understanding and advanced reasoning\n- Runs on any OpenAI-compatible LLM backend (results vary by model)\n- GPU-accelerated embeddings with auto CUDA/MPS/CPU detection\n- No runtime downloads — models pre-cached locally\n\n## 📦 Installation\n\n1. **Clone the repository**\n   ```bash\n   git clone https://github.com/Angad2005/ChatPDF.git\n   cd ChatPDF\n   ```\n\n2. **Create and activate a virtual environment** (recommended)\n   ```bash\n   python3 -m venv van1\n   source van1/bin/activate      # Linux/Mac\n   # .\\van1\\Scripts\\activate     # Windows\n   ```\n\n3. **Install dependencies**\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n4. **Install PyTorch with CUDA support** (for GPU embeddings)\n   ```bash\n   pip install torch --index-url https://download.pytorch.org/whl/cu121\n   ```\n\n5. **Pre-download the embedding model** (one-time, requires internet)\n   ```bash\n   python -c \"from sentence_transformers import SentenceTransformer; SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')\"\n   ```\n   This caches `~90 MB` to `~/.cache/huggingface/hub/`. Set `HF_HOME` to change cache location.\n\n## 🐳 Usage\n\n### Step 1: Start the MCP Summarization Service (optional, for summary generation)\n```bash\n# In a separate terminal:\nuvicorn mcp_main:app --host 127.0.0.1 --port 8001\n```\n\n### Step 2: Start the Streamlit Application\n```bash\n# Disable file watcher to avoid transformers import errors\nSTREAMLIT_SERVER_FILE_WATCHER_TYPE=none streamlit run main.py\n```\nThe application opens at `http://localhost:8501`.\n\n### Step 3: Use the Application\n\n1. **Configure LLM** (Sidebar → 🤖 LLM Settings):\n   - **API Base URL**: `http://localhost:1234/v1` (LM Studio), `http://localhost:8000/v1` (vLLM), `http://localhost:11434/v1` (Ollama), `https://integrate.api.nvidia.com/v1` (NVIDIA NIM)\n   - **API Key**: `not-needed` (local) or your API key (cloud)\n   - **Model Name**: Exact model ID from your server (e.g., `llama-3.2-1b-instruct`, `meta/llama-3.1-8b-instruct`)\n   - Click **💾 Save & Connect** — tests `/v1/models` endpoint\n\n2. **Load Embedding Model** (Sidebar → 📦 Embedding Model):\n   - Click **📂 Load from Cache** — loads `sentence-transformers/all-MiniLM-L6-v2` from local cache\n   - Auto-detects GPU (CUDA/MPS) with CPU fallback\n   - Shows cache path and device in UI\n\n3. **Upload Documents**: PDF, DOCX, TXT via sidebar file uploader\n\n4. **Ask Questions**: Chat with your documents in the main area\n\n5. **Generate Summaries** (requires MCP service):\n   - Select format (PDF/DOCX)\n   - Click **Create Summary Document**\n   - Download the result\n\n6. **Reset**: **🗑️ Clear Chat** clears history and vector store\n\n## ⚙️ Configuration\n\n### LLM Backend (via Sidebar GUI — no code edits needed)\n\n| Backend | API Base URL | API Key | Example Model Name |\n|---------|--------------|---------|-------------------|\n| LM Studio | `http://localhost:1234/v1` | `not-needed` | `llama-3.2-1b-instruct` |\n| vLLM | `http://localhost:8000/v1` | `not-needed` | `meta-llama/Llama-3.1-8B-Instruct` |\n| Ollama | `http://localhost:11434/v1` | `not-needed` | `llama3.1:8b` |\n| NVIDIA NIM (cloud) | `https://integrate.api.nvidia.com/v1` | your NVIDIA API key | `meta/llama-3.1-8b-instruct` |\n| NVIDIA NIM (local) | `http://localhost:8000/v1` | `not-needed` | `meta/llama-3.1-8b-instruct` |\n| OpenAI | `https://api.openai.com/v1` | `sk-...` | `gpt-4o-mini` |\n| Together AI | `https://api.together.xyz/v1` | your key | `meta-llama/Llama-3.1-8B-Instruct-Turbo` |\n\nThe server must implement:\n- `GET /v1/models` — returns `{ \"data\": [{ \"id\": \"model-name\" }, ...] }`\n- `POST /v1/chat/completions` — OpenAI chat completion format\n\n### Environment Variables (Optional)\n```bash\n# Embedding model cache location\nexport HF_HOME=/path/to/cache\n\n# Hugging Face mirror (if blocked)\nexport HF_ENDPOINT=https://hf-mirror.com\n\n# LLM defaults (overridden by sidebar)\nexport LLM_API_BASE=http://localhost:1234/v1\nexport LLM_API_KEY=not-needed\nexport LLM_MODEL_NAME=llama-3.2-1b-instruct\n```\n\n## 📄 File Format Support\n- PDF (.pdf)\n- Word Document (.docx)\n- Plain Text (.txt)\n\n## 🔧 Troubleshooting\n\n### Streamlit file watcher errors (transformers/torch import)\n**Fixed by:** `STREAMLIT_SERVER_FILE_WATCHER_TYPE=none streamlit run main.py`\n\n### Embedding model not found in cache\nRun the pre-download command from Installation step 5. The UI shows the cache path it's checking.\n\n### GPU not used\n- Verify CUDA: `python -c \"import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))\"`\n- Reinstall PyTorch with CUDA: `pip install torch --index-url https://download.pytorch.org/whl/cu121`\n- The app auto-detects CUDA → MPS → CPU\n\n### LLM connection fails\n- Test endpoint: `curl http://localhost:1234/v1/models`\n- Check model name matches exactly what `/v1/models` returns\n- For LM Studio: Enable \"OpenAI Compatible Server\" in Settings → Developer\n\n### SSL/Cert errors downloading model\n```bash\npip install --upgrade certifi\n# Or use mirror:\nHF_ENDPOINT=https://hf-mirror.com python -c \"from sentence_transformers import SentenceTransformer; SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')\"\n```\n\n## 📝 License\nMIT License — see [LICENSE](LICENSE) for details.\n\n## 🙏 Acknowledgments\n- [Streamlit](https://streamlit.io/)\n- [LangChain](https://www.langchain.com/)\n- [Hugging Face Transformers](https://huggingface.co/docs/transformers/index)\n- [sentence-transformers](https://www.sbert.net/)\n- [FastAPI](https://fastapi.tiangolo.com/)\n- [PyTorch](https://pytorch.org/)"}}}}