1 line
22 KiB
JSON
1 line
22 KiB
JSON
{"repo": "agisota/pzdrk-ext", "files": {"REFACTOR-PLAN.md": {"926764517cad9d069ac31f4d40718b0029367a31": {"t": 1785747170.454877, "n": 9316, "h": "255a1709da035640", "c": "# PZDRK v7.0 — COMPREHENSIVE REFACTOR PLAN\n\n## Executive Summary\n\nНа основе глубокого анализа текущей кодовой базы (5,006 строк content.js, монолитная архитектура) предлагается комплексная реорганизация системы для достижения **20-200x ускорения** и централизации данных.\n\n---\n\n## 🎯 Целевые метрики\n\n| Метрика | Текущее | Цель | Как |\n|---------|---------|------|-----|\n| **Latency (summary)** | 2-8 сек | 0.3-1 сек | Racing execution, caching |\n| **Throughput** | 12 req/min | 200+ req/min | Parallel processing, 10+ keys |\n| **Availability** | Single provider | 99.9% | 3+ providers, circuit breakers |\n| **Data persistence** | Manual download | Auto sync everywhere | Milvus + Obsidian pipeline |\n| **Search** | None | Semantic | Vector embeddings |\n\n---\n\n## 🏗️ Архитектура\n\n### Current (Anti-pattern)\n```\ncontent.js (5,006 lines)\n├── UI rendering\n├── API calls\n├── Business logic\n├── State management\n└── Caching\n```\n\n### Target (Modular)\n```\nsrc/\n├── core/\n│ ├── api-router/ # Racing, circuit breakers\n│ ├── key-manager/ # 10+ keys per provider\n│ ├── cache-layer/ # Multi-tier\n│ └── vector-store/ # Milvus client\n├── content/\n│ ├── ui/ # Notes, palette, mindmap\n│ ├── extraction/ # Page scraping\n│ └── handlers/ # Message handlers\n├── background/\n│ ├── service-worker/\n│ └── api-proxies/\n├── storage/\n│ ├── obsidian-sync/\n│ ├── milvus-client/\n│ └── local-indexeddb/\n└── shared/\n ├── types/ # TypeScript\n ├── constants/\n └── utils/\n\nserver/ # Local data hub\n├── index.ts # Fastify + WS\n├── routes/\n└── sync/\n\ndocker-compose.milvus.yml # Vector DB\n```\n\n---\n\n## ⚡ PHASE 1: Ultra-Performance API Layer\n\n### 1.1 Smart API Router\n\n**Racing Execution** — запускаем запрос на 3 провайдера одновременно:\n\n```typescript\n// Было: await callGroq(prompt) — 2-5 сек\n// Стало: Promise.race([groq, cerebras, openai]) — 0.5-1 сек\n\nasync executeWithRacing(request): Promise<ApiResponse> {\n const candidates = [groq, cerebras, openai];\n const controllers = candidates.map(() => new AbortController());\n \n const promises = candidates.map(async (provider, i) => {\n const result = await this.call(provider, request, controllers[i].signal);\n // Отменяем остальные\n controllers.forEach((c, idx) => idx !== i && c.abort());\n return result;\n });\n\n return Promise.race(promises);\n}\n```\n\n**Circuit Breaker** — отключаем \"больных\" провайдеров:\n```typescript\nif (failureCount >= 5) {\n breaker.state = 'open';\n setTimeout(() => breaker.state = 'half-open', 30000);\n}\n```\n\n### 1.2 Key Rotation Engine\n\n**Weighted Selection** — выбираем лучший ключ на основе:\n- `remainingQuota` — сколько осталось\n- `avgLatency` — историческая скорость\n- `failureCount` — недавние ошибки\n- `timeSinceUse` — равномерное распределение\n\n```typescript\nprivate calculateKeyWeight(state: KeyState): number {\n const quotaFactor = state.remainingQuota / 10000;\n const latencyFactor = 1000 / (state.avgLatency + 100);\n const healthFactor = 1 / (state.failureCount + 1);\n const recencyFactor = 1 + timeSinceUse / 60000;\n \n return quotaFactor * latencyFactor * healthFactor * recencyFactor;\n}\n```\n\n**Поддерживаемые провайдеры:**\n- Groq (primary) — 10+ ключей\n- Cerebras (fallback) — 5+ ключей \n- OpenAI (backup) — 3+ ключей\n- Anthropic (optional)\n- Google (optional)\n- DeepSeek (optional)\n\n---\n\n## 🧠 PHASE 2: Vector Database (Milvus)\n\n### 2.1 Self-Hosted Setup\n\n```yaml\n# docker-compose.milvus.yml\nservices:\n milvus-standalone:\n image: milvusdb/milvus:v2.3.3\n ports:\n - \"19530:19530\" # gRPC API\n \n attu: # GUI\n image: zilliz/attu:v2.3.5\n ports:\n - \"8000:3000\"\n```\n\n### 2.2 Document Indexing\n\n```typescript\n// Auto-chunking с respect for boundaries\nconst chunks = MilvusVectorStore.smartChunk(content, {\n chunkSize: 512,\n overlap: 128,\n respectBoundaries: true // Не режем предложения\n});\n\n// Generate embeddings (OpenAI text-embedding-3-small)\nconst embeddings = await Promise.all(\n chunks.map(c => embed(c))\n);\n\n// Store in Milvus\nawait milvus.indexDocument({\n id: `${url}-${i}`,\n vector: embeddings[i],\n metadata: { url, title, timestamp, tags }\n});\n```\n\n### 2.3 Semantic Search\n\n```typescript\n// Найти похожие страницы\nconst results = await milvus.searchSimilar(query, embedding, {\n topK: 5,\n minScore: 0.7,\n filter: { \n domains: ['github.com', 'docs.python.org'],\n dateFrom: new Date('2024-01-01')\n }\n});\n```\n\n---\n\n## 📝 PHASE 3: Obsidian Integration\n\n### 3.1 Two-Way Sync\n\n```typescript\nclass ObsidianSyncManager {\n // Save from browser → Obsidian\n async saveSummary(summary: SummaryData): Promise<void> {\n const content = this.renderTemplate(summary);\n await this.atomicWrite(filepath, content);\n }\n\n // Watch for changes in Obsidian\n startWatcher() {\n chokidar.watch(vaultPath).on('change', (file) => {\n const summary = this.parseNote(file);\n this.emitChange(summary); // Sync back to extension\n });\n }\n}\n```\n\n### 3.2 Note Template\n\n```markdown\n---\ncreated: 2026-02-12T10:30:00Z\nurl: https://...\ndomain: github.com\ndepth: 4\ntags: [#llm, #architecture]\ntokens: 2450\n---\n\n# {{title}}\n\n> **URL**: [{{url}}]({{url}})\n> **Domain**: {{domain}} | **Depth**: {{depth}}/5\n\n## Суть\n{{core}}\n\n## Ключевые тезисы\n{{keyPoints}}\n\n## TIL\n{{til}}\n\n## Действия\n{{actions}}\n\n## Сущности\n{{entities}}\n\n---\n*Generated by pzdrk* [[pzdrk-index]]\n```\n\n---\n\n## 🌐 PHASE 4: Local Data Hub\n\n### 4.1 REST API (localhost:7420)\n\n```typescript\n// GET /api/summaries?page=1&limit=50&domain=github.com\n// GET /api/search (semantic)\n// POST /api/summaries (create/update)\n// GET /api/stats\n// WS /ws (real-time updates)\n```\n\n### 4.2 Sync Pipeline\n\n```\nBrowser Extension\n ↓\n Summary Created\n ↓\n ┌────┴────┐\n ↓ ↓\nMilvus Obsidian\n(Vector) (Files)\n ↓ ↓\n └────┬────┘\n ↓\n WebSocket Broadcast\n ↓\n Dashboard UI\n```\n\n---\n\n## 🚀 PHASE 5: Performance Optimizations\n\n### 5.1 Streaming & Incremental Rendering\n\n```typescript\nasync renderStreaming(stream: ReadableStream, element: HTMLElement) {\n const reader = stream.getReader();\n const parser = new StreamingJsonParser();\n \n while (true) {\n const { done, value } = await reader.read();\n if (done) break;\n \n const partial = parser.parseIncremental(value);\n this.updateDOM(element, partial);\n await this.frameThrottle();\n }\n}\n```\n\n### 5.2 Predictive Prefetching\n\n```typescript\nclass PredictivePrefetchEngine {\n async predictNextPages(currentUrl: string): Promise<Prediction[]> {\n // ML-based prediction на основе:\n // - Истории переходов\n // - Ссылок на странице\n // - Времени дня\n return this.userModel.predictNextUrls(features, 5);\n }\n\n async startPrefetching() {\n const predictions = await this.predictNextPages(location.href);\n \n for (const pred of predictions) {\n scheduler.postTask(async () => {\n const summary = await this.generateSummary(pred.url);\n this.cache.set(pred.url, summary);\n }, { priority: 'background' });\n }\n }\n}\n```\n\n---\n\n## 📦 Installation & Setup\n\n### Prerequisites\n```bash\n# macOS\nbrew install docker docker-compose node\n\n# Start Docker Desktop\nopen -a Docker\n```\n\n### Quick Start\n```bash\n# 1. Clone and setup\ncd /Users/marklindgreen/Gronxie\\ Dropbox/MARK\\ HAGGERTY/_DEV/_cli/ext\nnpm install\n\n# 2. Start Milvus\ndocker-compose -f docker-compose.milvus.yml up -d\n\n# 3. Configure keys\ncat > .env << EOF\nGROQ_API_KEYS=gsk_key1,gsk_key2,gsk_key3,gsk_key4,gsk_key5\nCEREBRAS_API_KEYS=csk_key1,csk_key2,csk_key3\nOPENAI_API_KEY=sk-...\nOBSIDIAN_VAULT_PATH=/Users/.../Documents/Obsidian/Main\nEOF\n\n# 4. Start data hub\nnpm run server\n\n# 5. Build extension\nnpm run build\n\n# 6. Load in Chrome\n# chrome://extensions → Developer mode → Load unpacked → dist/\n```\n\n---\n\n## 📊 Monitoring & Debugging\n\n### Key Metrics\n```typescript\n// GET /api/metrics\n{\n \"providers\": {\n \"groq\": { \"availableKeys\": 8, \"avgLatency\": 450, \"requestsPerMinute\": 45 },\n \"cerebras\": { \"availableKeys\": 3, \"avgLatency\": 720, \"requestsPerMinute\": 12 }\n },\n \"cache\": { \"hitRate\": 0.34, \"size\": \"145MB\" },\n \"milvus\": { \"totalDocuments\": 1250, \"indexSize\": \"2.3GB\" }\n}\n```\n\n### Attu GUI\nОткрыть http://localhost:8000 для просмотра:\n- Коллекций\n- Векторов\n- Query performance\n\n---\n\n## 🗓️ Roadmap\n\n| Phase | Duration | Key Deliverables |\n|-------|----------|------------------|\n| 1 | Week 1-2 | TypeScript + Vite, Smart Router |\n| 2 | Week 3-4 | Milvus integration, Key Rotation |\n| 3 | Week 5-6 | Obsidian sync, Local API |\n| 4 | Week 7-8 | Streaming, Prefetching |\n| 5 | Week 9-10 | Dashboard, CLI tools |\n| 6 | Week 11-12 | Testing, docs, release |\n\n---\n\n## 🔐 Security Considerations\n\n1. **API Keys** — хранятся в `chrome.storage.sync`, никогда не логируются\n2. **Local Milvus** — только localhost, нет external access\n3. **Obsidian Vault** — file system access через разрешения пользователя\n4. **Data retention** — пользователь контролирует все данные локально\n\n---\n\n## 💡 Future Enhancements\n\n- **Local LLM** — интеграция с Ollama/LM Studio для offline mode\n- **Graph RAG** — knowledge graph на базе страниц\n- **Collaborative** — sync между устройствами через облако\n- **Mobile app** — companion app для iOS/Android\n\n---\n\n*Generated for pzdrk v7.0 Advanced Architecture*\n"}, "134379bc6c4063e0507847def0fa228ffbbb25aa": {"t": 1785770925.275587, "n": 9316, "h": "255a1709da035640", "c": "# PZDRK v7.0 — COMPREHENSIVE REFACTOR PLAN\n\n## Executive Summary\n\nНа основе глубокого анализа текущей кодовой базы (5,006 строк content.js, монолитная архитектура) предлагается комплексная реорганизация системы для достижения **20-200x ускорения** и централизации данных.\n\n---\n\n## 🎯 Целевые метрики\n\n| Метрика | Текущее | Цель | Как |\n|---------|---------|------|-----|\n| **Latency (summary)** | 2-8 сек | 0.3-1 сек | Racing execution, caching |\n| **Throughput** | 12 req/min | 200+ req/min | Parallel processing, 10+ keys |\n| **Availability** | Single provider | 99.9% | 3+ providers, circuit breakers |\n| **Data persistence** | Manual download | Auto sync everywhere | Milvus + Obsidian pipeline |\n| **Search** | None | Semantic | Vector embeddings |\n\n---\n\n## 🏗️ Архитектура\n\n### Current (Anti-pattern)\n```\ncontent.js (5,006 lines)\n├── UI rendering\n├── API calls\n├── Business logic\n├── State management\n└── Caching\n```\n\n### Target (Modular)\n```\nsrc/\n├── core/\n│ ├── api-router/ # Racing, circuit breakers\n│ ├── key-manager/ # 10+ keys per provider\n│ ├── cache-layer/ # Multi-tier\n│ └── vector-store/ # Milvus client\n├── content/\n│ ├── ui/ # Notes, palette, mindmap\n│ ├── extraction/ # Page scraping\n│ └── handlers/ # Message handlers\n├── background/\n│ ├── service-worker/\n│ └── api-proxies/\n├── storage/\n│ ├── obsidian-sync/\n│ ├── milvus-client/\n│ └── local-indexeddb/\n└── shared/\n ├── types/ # TypeScript\n ├── constants/\n └── utils/\n\nserver/ # Local data hub\n├── index.ts # Fastify + WS\n├── routes/\n└── sync/\n\ndocker-compose.milvus.yml # Vector DB\n```\n\n---\n\n## ⚡ PHASE 1: Ultra-Performance API Layer\n\n### 1.1 Smart API Router\n\n**Racing Execution** — запускаем запрос на 3 провайдера одновременно:\n\n```typescript\n// Было: await callGroq(prompt) — 2-5 сек\n// Стало: Promise.race([groq, cerebras, openai]) — 0.5-1 сек\n\nasync executeWithRacing(request): Promise<ApiResponse> {\n const candidates = [groq, cerebras, openai];\n const controllers = candidates.map(() => new AbortController());\n \n const promises = candidates.map(async (provider, i) => {\n const result = await this.call(provider, request, controllers[i].signal);\n // Отменяем остальные\n controllers.forEach((c, idx) => idx !== i && c.abort());\n return result;\n });\n\n return Promise.race(promises);\n}\n```\n\n**Circuit Breaker** — отключаем \"больных\" провайдеров:\n```typescript\nif (failureCount >= 5) {\n breaker.state = 'open';\n setTimeout(() => breaker.state = 'half-open', 30000);\n}\n```\n\n### 1.2 Key Rotation Engine\n\n**Weighted Selection** — выбираем лучший ключ на основе:\n- `remainingQuota` — сколько осталось\n- `avgLatency` — историческая скорость\n- `failureCount` — недавние ошибки\n- `timeSinceUse` — равномерное распределение\n\n```typescript\nprivate calculateKeyWeight(state: KeyState): number {\n const quotaFactor = state.remainingQuota / 10000;\n const latencyFactor = 1000 / (state.avgLatency + 100);\n const healthFactor = 1 / (state.failureCount + 1);\n const recencyFactor = 1 + timeSinceUse / 60000;\n \n return quotaFactor * latencyFactor * healthFactor * recencyFactor;\n}\n```\n\n**Поддерживаемые провайдеры:**\n- Groq (primary) — 10+ ключей\n- Cerebras (fallback) — 5+ ключей \n- OpenAI (backup) — 3+ ключей\n- Anthropic (optional)\n- Google (optional)\n- DeepSeek (optional)\n\n---\n\n## 🧠 PHASE 2: Vector Database (Milvus)\n\n### 2.1 Self-Hosted Setup\n\n```yaml\n# docker-compose.milvus.yml\nservices:\n milvus-standalone:\n image: milvusdb/milvus:v2.3.3\n ports:\n - \"19530:19530\" # gRPC API\n \n attu: # GUI\n image: zilliz/attu:v2.3.5\n ports:\n - \"8000:3000\"\n```\n\n### 2.2 Document Indexing\n\n```typescript\n// Auto-chunking с respect for boundaries\nconst chunks = MilvusVectorStore.smartChunk(content, {\n chunkSize: 512,\n overlap: 128,\n respectBoundaries: true // Не режем предложения\n});\n\n// Generate embeddings (OpenAI text-embedding-3-small)\nconst embeddings = await Promise.all(\n chunks.map(c => embed(c))\n);\n\n// Store in Milvus\nawait milvus.indexDocument({\n id: `${url}-${i}`,\n vector: embeddings[i],\n metadata: { url, title, timestamp, tags }\n});\n```\n\n### 2.3 Semantic Search\n\n```typescript\n// Найти похожие страницы\nconst results = await milvus.searchSimilar(query, embedding, {\n topK: 5,\n minScore: 0.7,\n filter: { \n domains: ['github.com', 'docs.python.org'],\n dateFrom: new Date('2024-01-01')\n }\n});\n```\n\n---\n\n## 📝 PHASE 3: Obsidian Integration\n\n### 3.1 Two-Way Sync\n\n```typescript\nclass ObsidianSyncManager {\n // Save from browser → Obsidian\n async saveSummary(summary: SummaryData): Promise<void> {\n const content = this.renderTemplate(summary);\n await this.atomicWrite(filepath, content);\n }\n\n // Watch for changes in Obsidian\n startWatcher() {\n chokidar.watch(vaultPath).on('change', (file) => {\n const summary = this.parseNote(file);\n this.emitChange(summary); // Sync back to extension\n });\n }\n}\n```\n\n### 3.2 Note Template\n\n```markdown\n---\ncreated: 2026-02-12T10:30:00Z\nurl: https://...\ndomain: github.com\ndepth: 4\ntags: [#llm, #architecture]\ntokens: 2450\n---\n\n# {{title}}\n\n> **URL**: [{{url}}]({{url}})\n> **Domain**: {{domain}} | **Depth**: {{depth}}/5\n\n## Суть\n{{core}}\n\n## Ключевые тезисы\n{{keyPoints}}\n\n## TIL\n{{til}}\n\n## Действия\n{{actions}}\n\n## Сущности\n{{entities}}\n\n---\n*Generated by pzdrk* [[pzdrk-index]]\n```\n\n---\n\n## 🌐 PHASE 4: Local Data Hub\n\n### 4.1 REST API (localhost:7420)\n\n```typescript\n// GET /api/summaries?page=1&limit=50&domain=github.com\n// GET /api/search (semantic)\n// POST /api/summaries (create/update)\n// GET /api/stats\n// WS /ws (real-time updates)\n```\n\n### 4.2 Sync Pipeline\n\n```\nBrowser Extension\n ↓\n Summary Created\n ↓\n ┌────┴────┐\n ↓ ↓\nMilvus Obsidian\n(Vector) (Files)\n ↓ ↓\n └────┬────┘\n ↓\n WebSocket Broadcast\n ↓\n Dashboard UI\n```\n\n---\n\n## 🚀 PHASE 5: Performance Optimizations\n\n### 5.1 Streaming & Incremental Rendering\n\n```typescript\nasync renderStreaming(stream: ReadableStream, element: HTMLElement) {\n const reader = stream.getReader();\n const parser = new StreamingJsonParser();\n \n while (true) {\n const { done, value } = await reader.read();\n if (done) break;\n \n const partial = parser.parseIncremental(value);\n this.updateDOM(element, partial);\n await this.frameThrottle();\n }\n}\n```\n\n### 5.2 Predictive Prefetching\n\n```typescript\nclass PredictivePrefetchEngine {\n async predictNextPages(currentUrl: string): Promise<Prediction[]> {\n // ML-based prediction на основе:\n // - Истории переходов\n // - Ссылок на странице\n // - Времени дня\n return this.userModel.predictNextUrls(features, 5);\n }\n\n async startPrefetching() {\n const predictions = await this.predictNextPages(location.href);\n \n for (const pred of predictions) {\n scheduler.postTask(async () => {\n const summary = await this.generateSummary(pred.url);\n this.cache.set(pred.url, summary);\n }, { priority: 'background' });\n }\n }\n}\n```\n\n---\n\n## 📦 Installation & Setup\n\n### Prerequisites\n```bash\n# macOS\nbrew install docker docker-compose node\n\n# Start Docker Desktop\nopen -a Docker\n```\n\n### Quick Start\n```bash\n# 1. Clone and setup\ncd /Users/marklindgreen/Gronxie\\ Dropbox/MARK\\ HAGGERTY/_DEV/_cli/ext\nnpm install\n\n# 2. Start Milvus\ndocker-compose -f docker-compose.milvus.yml up -d\n\n# 3. Configure keys\ncat > .env << EOF\nGROQ_API_KEYS=gsk_key1,gsk_key2,gsk_key3,gsk_key4,gsk_key5\nCEREBRAS_API_KEYS=csk_key1,csk_key2,csk_key3\nOPENAI_API_KEY=sk-...\nOBSIDIAN_VAULT_PATH=/Users/.../Documents/Obsidian/Main\nEOF\n\n# 4. Start data hub\nnpm run server\n\n# 5. Build extension\nnpm run build\n\n# 6. Load in Chrome\n# chrome://extensions → Developer mode → Load unpacked → dist/\n```\n\n---\n\n## 📊 Monitoring & Debugging\n\n### Key Metrics\n```typescript\n// GET /api/metrics\n{\n \"providers\": {\n \"groq\": { \"availableKeys\": 8, \"avgLatency\": 450, \"requestsPerMinute\": 45 },\n \"cerebras\": { \"availableKeys\": 3, \"avgLatency\": 720, \"requestsPerMinute\": 12 }\n },\n \"cache\": { \"hitRate\": 0.34, \"size\": \"145MB\" },\n \"milvus\": { \"totalDocuments\": 1250, \"indexSize\": \"2.3GB\" }\n}\n```\n\n### Attu GUI\nОткрыть http://localhost:8000 для просмотра:\n- Коллекций\n- Векторов\n- Query performance\n\n---\n\n## 🗓️ Roadmap\n\n| Phase | Duration | Key Deliverables |\n|-------|----------|------------------|\n| 1 | Week 1-2 | TypeScript + Vite, Smart Router |\n| 2 | Week 3-4 | Milvus integration, Key Rotation |\n| 3 | Week 5-6 | Obsidian sync, Local API |\n| 4 | Week 7-8 | Streaming, Prefetching |\n| 5 | Week 9-10 | Dashboard, CLI tools |\n| 6 | Week 11-12 | Testing, docs, release |\n\n---\n\n## 🔐 Security Considerations\n\n1. **API Keys** — хранятся в `chrome.storage.sync`, никогда не логируются\n2. **Local Milvus** — только localhost, нет external access\n3. **Obsidian Vault** — file system access через разрешения пользователя\n4. **Data retention** — пользователь контролирует все данные локально\n\n---\n\n## 💡 Future Enhancements\n\n- **Local LLM** — интеграция с Ollama/LM Studio для offline mode\n- **Graph RAG** — knowledge graph на базе страниц\n- **Collaborative** — sync между устройствами через облако\n- **Mobile app** — companion app для iOS/Android\n\n---\n\n*Generated for pzdrk v7.0 Advanced Architecture*\n"}}}} |