{"repo": "agisota/pzdrk-ext", "files": {"REFACTOR-PLAN.md": {"926764517cad9d069ac31f4d40718b0029367a31": {"t": 1785747170.454877, "n": 9316, "h": "255a1709da035640", "c": "# PZDRK v7.0 — COMPREHENSIVE REFACTOR PLAN\n\n## Executive Summary\n\nНа основе глубокого анализа текущей кодовой базы (5,006 строк content.js, монолитная архитектура) предлагается комплексная реорганизация системы для достижения **20-200x ускорения** и централизации данных.\n\n---\n\n## 🎯 Целевые метрики\n\n| Метрика | Текущее | Цель | Как |\n|---------|---------|------|-----|\n| **Latency (summary)** | 2-8 сек | 0.3-1 сек | Racing execution, caching |\n| **Throughput** | 12 req/min | 200+ req/min | Parallel processing, 10+ keys |\n| **Availability** | Single provider | 99.9% | 3+ providers, circuit breakers |\n| **Data persistence** | Manual download | Auto sync everywhere | Milvus + Obsidian pipeline |\n| **Search** | None | Semantic | Vector embeddings |\n\n---\n\n## 🏗️ Архитектура\n\n### Current (Anti-pattern)\n```\ncontent.js (5,006 lines)\n├── UI rendering\n├── API calls\n├── Business logic\n├── State management\n└── Caching\n```\n\n### Target (Modular)\n```\nsrc/\n├── core/\n│   ├── api-router/          # Racing, circuit breakers\n│   ├── key-manager/         # 10+ keys per provider\n│   ├── cache-layer/         # Multi-tier\n│   └── vector-store/        # Milvus client\n├── content/\n│   ├── ui/                  # Notes, palette, mindmap\n│   ├── extraction/          # Page scraping\n│   └── handlers/            # Message handlers\n├── background/\n│   ├── service-worker/\n│   └── api-proxies/\n├── storage/\n│   ├── obsidian-sync/\n│   ├── milvus-client/\n│   └── local-indexeddb/\n└── shared/\n    ├── types/               # TypeScript\n    ├── constants/\n    └── utils/\n\nserver/                      # Local data hub\n├── index.ts                 # Fastify + WS\n├── routes/\n└── sync/\n\ndocker-compose.milvus.yml    # Vector DB\n```\n\n---\n\n## ⚡ PHASE 1: Ultra-Performance API Layer\n\n### 1.1 Smart API Router\n\n**Racing Execution** — запускаем запрос на 3 провайдера одновременно:\n\n```typescript\n// Было: await callGroq(prompt) — 2-5 сек\n// Стало: Promise.race([groq, cerebras, openai]) — 0.5-1 сек\n\nasync executeWithRacing(request): Promise<ApiResponse> {\n  const candidates = [groq, cerebras, openai];\n  const controllers = candidates.map(() => new AbortController());\n  \n  const promises = candidates.map(async (provider, i) => {\n    const result = await this.call(provider, request, controllers[i].signal);\n    // Отменяем остальные\n    controllers.forEach((c, idx) => idx !== i && c.abort());\n    return result;\n  });\n\n  return Promise.race(promises);\n}\n```\n\n**Circuit Breaker** — отключаем \"больных\" провайдеров:\n```typescript\nif (failureCount >= 5) {\n  breaker.state = 'open';\n  setTimeout(() => breaker.state = 'half-open', 30000);\n}\n```\n\n### 1.2 Key Rotation Engine\n\n**Weighted Selection** — выбираем лучший ключ на основе:\n- `remainingQuota` — сколько осталось\n- `avgLatency` — историческая скорость\n- `failureCount` — недавние ошибки\n- `timeSinceUse` — равномерное распределение\n\n```typescript\nprivate calculateKeyWeight(state: KeyState): number {\n  const quotaFactor = state.remainingQuota / 10000;\n  const latencyFactor = 1000 / (state.avgLatency + 100);\n  const healthFactor = 1 / (state.failureCount + 1);\n  const recencyFactor = 1 + timeSinceUse / 60000;\n  \n  return quotaFactor * latencyFactor * healthFactor * recencyFactor;\n}\n```\n\n**Поддерживаемые провайдеры:**\n- Groq (primary) — 10+ ключей\n- Cerebras (fallback) — 5+ ключей  \n- OpenAI (backup) — 3+ ключей\n- Anthropic (optional)\n- Google (optional)\n- DeepSeek (optional)\n\n---\n\n## 🧠 PHASE 2: Vector Database (Milvus)\n\n### 2.1 Self-Hosted Setup\n\n```yaml\n# docker-compose.milvus.yml\nservices:\n  milvus-standalone:\n    image: milvusdb/milvus:v2.3.3\n    ports:\n      - \"19530:19530\"    # gRPC API\n  \n  attu:                    # GUI\n    image: zilliz/attu:v2.3.5\n    ports:\n      - \"8000:3000\"\n```\n\n### 2.2 Document Indexing\n\n```typescript\n// Auto-chunking с respect for boundaries\nconst chunks = MilvusVectorStore.smartChunk(content, {\n  chunkSize: 512,\n  overlap: 128,\n  respectBoundaries: true  // Не режем предложения\n});\n\n// Generate embeddings (OpenAI text-embedding-3-small)\nconst embeddings = await Promise.all(\n  chunks.map(c => embed(c))\n);\n\n// Store in Milvus\nawait milvus.indexDocument({\n  id: `${url}-${i}`,\n  vector: embeddings[i],\n  metadata: { url, title, timestamp, tags }\n});\n```\n\n### 2.3 Semantic Search\n\n```typescript\n// Найти похожие страницы\nconst results = await milvus.searchSimilar(query, embedding, {\n  topK: 5,\n  minScore: 0.7,\n  filter: { \n    domains: ['github.com', 'docs.python.org'],\n    dateFrom: new Date('2024-01-01')\n  }\n});\n```\n\n---\n\n## 📝 PHASE 3: Obsidian Integration\n\n### 3.1 Two-Way Sync\n\n```typescript\nclass ObsidianSyncManager {\n  // Save from browser → Obsidian\n  async saveSummary(summary: SummaryData): Promise<void> {\n    const content = this.renderTemplate(summary);\n    await this.atomicWrite(filepath, content);\n  }\n\n  // Watch for changes in Obsidian\n  startWatcher() {\n    chokidar.watch(vaultPath).on('change', (file) => {\n      const summary = this.parseNote(file);\n      this.emitChange(summary);  // Sync back to extension\n    });\n  }\n}\n```\n\n### 3.2 Note Template\n\n```markdown\n---\ncreated: 2026-02-12T10:30:00Z\nurl: https://...\ndomain: github.com\ndepth: 4\ntags: [#llm, #architecture]\ntokens: 2450\n---\n\n# {{title}}\n\n> **URL**: [{{url}}]({{url}})\n> **Domain**: {{domain}} | **Depth**: {{depth}}/5\n\n## Суть\n{{core}}\n\n## Ключевые тезисы\n{{keyPoints}}\n\n## TIL\n{{til}}\n\n## Действия\n{{actions}}\n\n## Сущности\n{{entities}}\n\n---\n*Generated by pzdrk* [[pzdrk-index]]\n```\n\n---\n\n## 🌐 PHASE 4: Local Data Hub\n\n### 4.1 REST API (localhost:7420)\n\n```typescript\n// GET /api/summaries?page=1&limit=50&domain=github.com\n// GET /api/search (semantic)\n// POST /api/summaries (create/update)\n// GET /api/stats\n// WS /ws (real-time updates)\n```\n\n### 4.2 Sync Pipeline\n\n```\nBrowser Extension\n       ↓\n   Summary Created\n       ↓\n  ┌────┴────┐\n  ↓         ↓\nMilvus   Obsidian\n(Vector)  (Files)\n  ↓         ↓\n  └────┬────┘\n       ↓\n  WebSocket Broadcast\n       ↓\n  Dashboard UI\n```\n\n---\n\n## 🚀 PHASE 5: Performance Optimizations\n\n### 5.1 Streaming & Incremental Rendering\n\n```typescript\nasync renderStreaming(stream: ReadableStream, element: HTMLElement) {\n  const reader = stream.getReader();\n  const parser = new StreamingJsonParser();\n  \n  while (true) {\n    const { done, value } = await reader.read();\n    if (done) break;\n    \n    const partial = parser.parseIncremental(value);\n    this.updateDOM(element, partial);\n    await this.frameThrottle();\n  }\n}\n```\n\n### 5.2 Predictive Prefetching\n\n```typescript\nclass PredictivePrefetchEngine {\n  async predictNextPages(currentUrl: string): Promise<Prediction[]> {\n    // ML-based prediction на основе:\n    // - Истории переходов\n    // - Ссылок на странице\n    // - Времени дня\n    return this.userModel.predictNextUrls(features, 5);\n  }\n\n  async startPrefetching() {\n    const predictions = await this.predictNextPages(location.href);\n    \n    for (const pred of predictions) {\n      scheduler.postTask(async () => {\n        const summary = await this.generateSummary(pred.url);\n        this.cache.set(pred.url, summary);\n      }, { priority: 'background' });\n    }\n  }\n}\n```\n\n---\n\n## 📦 Installation & Setup\n\n### Prerequisites\n```bash\n# macOS\nbrew install docker docker-compose node\n\n# Start Docker Desktop\nopen -a Docker\n```\n\n### Quick Start\n```bash\n# 1. Clone and setup\ncd /Users/marklindgreen/Gronxie\\ Dropbox/MARK\\ HAGGERTY/_DEV/_cli/ext\nnpm install\n\n# 2. Start Milvus\ndocker-compose -f docker-compose.milvus.yml up -d\n\n# 3. Configure keys\ncat > .env << EOF\nGROQ_API_KEYS=gsk_key1,gsk_key2,gsk_key3,gsk_key4,gsk_key5\nCEREBRAS_API_KEYS=csk_key1,csk_key2,csk_key3\nOPENAI_API_KEY=sk-...\nOBSIDIAN_VAULT_PATH=/Users/.../Documents/Obsidian/Main\nEOF\n\n# 4. Start data hub\nnpm run server\n\n# 5. Build extension\nnpm run build\n\n# 6. Load in Chrome\n# chrome://extensions → Developer mode → Load unpacked → dist/\n```\n\n---\n\n## 📊 Monitoring & Debugging\n\n### Key Metrics\n```typescript\n// GET /api/metrics\n{\n  \"providers\": {\n    \"groq\": { \"availableKeys\": 8, \"avgLatency\": 450, \"requestsPerMinute\": 45 },\n    \"cerebras\": { \"availableKeys\": 3, \"avgLatency\": 720, \"requestsPerMinute\": 12 }\n  },\n  \"cache\": { \"hitRate\": 0.34, \"size\": \"145MB\" },\n  \"milvus\": { \"totalDocuments\": 1250, \"indexSize\": \"2.3GB\" }\n}\n```\n\n### Attu GUI\nОткрыть http://localhost:8000 для просмотра:\n- Коллекций\n- Векторов\n- Query performance\n\n---\n\n## 🗓️ Roadmap\n\n| Phase | Duration | Key Deliverables |\n|-------|----------|------------------|\n| 1 | Week 1-2 | TypeScript + Vite, Smart Router |\n| 2 | Week 3-4 | Milvus integration, Key Rotation |\n| 3 | Week 5-6 | Obsidian sync, Local API |\n| 4 | Week 7-8 | Streaming, Prefetching |\n| 5 | Week 9-10 | Dashboard, CLI tools |\n| 6 | Week 11-12 | Testing, docs, release |\n\n---\n\n## 🔐 Security Considerations\n\n1. **API Keys** — хранятся в `chrome.storage.sync`, никогда не логируются\n2. **Local Milvus** — только localhost, нет external access\n3. **Obsidian Vault** — file system access через разрешения пользователя\n4. **Data retention** — пользователь контролирует все данные локально\n\n---\n\n## 💡 Future Enhancements\n\n- **Local LLM** — интеграция с Ollama/LM Studio для offline mode\n- **Graph RAG** — knowledge graph на базе страниц\n- **Collaborative** — sync между устройствами через облако\n- **Mobile app** — companion app для iOS/Android\n\n---\n\n*Generated for pzdrk v7.0 Advanced Architecture*\n"}, "134379bc6c4063e0507847def0fa228ffbbb25aa": {"t": 1785770925.275587, "n": 9316, "h": "255a1709da035640", "c": "# PZDRK v7.0 — COMPREHENSIVE REFACTOR PLAN\n\n## Executive Summary\n\nНа основе глубокого анализа текущей кодовой базы (5,006 строк content.js, монолитная архитектура) предлагается комплексная реорганизация системы для достижения **20-200x ускорения** и централизации данных.\n\n---\n\n## 🎯 Целевые метрики\n\n| Метрика | Текущее | Цель | Как |\n|---------|---------|------|-----|\n| **Latency (summary)** | 2-8 сек | 0.3-1 сек | Racing execution, caching |\n| **Throughput** | 12 req/min | 200+ req/min | Parallel processing, 10+ keys |\n| **Availability** | Single provider | 99.9% | 3+ providers, circuit breakers |\n| **Data persistence** | Manual download | Auto sync everywhere | Milvus + Obsidian pipeline |\n| **Search** | None | Semantic | Vector embeddings |\n\n---\n\n## 🏗️ Архитектура\n\n### Current (Anti-pattern)\n```\ncontent.js (5,006 lines)\n├── UI rendering\n├── API calls\n├── Business logic\n├── State management\n└── Caching\n```\n\n### Target (Modular)\n```\nsrc/\n├── core/\n│   ├── api-router/          # Racing, circuit breakers\n│   ├── key-manager/         # 10+ keys per provider\n│   ├── cache-layer/         # Multi-tier\n│   └── vector-store/        # Milvus client\n├── content/\n│   ├── ui/                  # Notes, palette, mindmap\n│   ├── extraction/          # Page scraping\n│   └── handlers/            # Message handlers\n├── background/\n│   ├── service-worker/\n│   └── api-proxies/\n├── storage/\n│   ├── obsidian-sync/\n│   ├── milvus-client/\n│   └── local-indexeddb/\n└── shared/\n    ├── types/               # TypeScript\n    ├── constants/\n    └── utils/\n\nserver/                      # Local data hub\n├── index.ts                 # Fastify + WS\n├── routes/\n└── sync/\n\ndocker-compose.milvus.yml    # Vector DB\n```\n\n---\n\n## ⚡ PHASE 1: Ultra-Performance API Layer\n\n### 1.1 Smart API Router\n\n**Racing Execution** — запускаем запрос на 3 провайдера одновременно:\n\n```typescript\n// Было: await callGroq(prompt) — 2-5 сек\n// Стало: Promise.race([groq, cerebras, openai]) — 0.5-1 сек\n\nasync executeWithRacing(request): Promise<ApiResponse> {\n  const candidates = [groq, cerebras, openai];\n  const controllers = candidates.map(() => new AbortController());\n  \n  const promises = candidates.map(async (provider, i) => {\n    const result = await this.call(provider, request, controllers[i].signal);\n    // Отменяем остальные\n    controllers.forEach((c, idx) => idx !== i && c.abort());\n    return result;\n  });\n\n  return Promise.race(promises);\n}\n```\n\n**Circuit Breaker** — отключаем \"больных\" провайдеров:\n```typescript\nif (failureCount >= 5) {\n  breaker.state = 'open';\n  setTimeout(() => breaker.state = 'half-open', 30000);\n}\n```\n\n### 1.2 Key Rotation Engine\n\n**Weighted Selection** — выбираем лучший ключ на основе:\n- `remainingQuota` — сколько осталось\n- `avgLatency` — историческая скорость\n- `failureCount` — недавние ошибки\n- `timeSinceUse` — равномерное распределение\n\n```typescript\nprivate calculateKeyWeight(state: KeyState): number {\n  const quotaFactor = state.remainingQuota / 10000;\n  const latencyFactor = 1000 / (state.avgLatency + 100);\n  const healthFactor = 1 / (state.failureCount + 1);\n  const recencyFactor = 1 + timeSinceUse / 60000;\n  \n  return quotaFactor * latencyFactor * healthFactor * recencyFactor;\n}\n```\n\n**Поддерживаемые провайдеры:**\n- Groq (primary) — 10+ ключей\n- Cerebras (fallback) — 5+ ключей  \n- OpenAI (backup) — 3+ ключей\n- Anthropic (optional)\n- Google (optional)\n- DeepSeek (optional)\n\n---\n\n## 🧠 PHASE 2: Vector Database (Milvus)\n\n### 2.1 Self-Hosted Setup\n\n```yaml\n# docker-compose.milvus.yml\nservices:\n  milvus-standalone:\n    image: milvusdb/milvus:v2.3.3\n    ports:\n      - \"19530:19530\"    # gRPC API\n  \n  attu:                    # GUI\n    image: zilliz/attu:v2.3.5\n    ports:\n      - \"8000:3000\"\n```\n\n### 2.2 Document Indexing\n\n```typescript\n// Auto-chunking с respect for boundaries\nconst chunks = MilvusVectorStore.smartChunk(content, {\n  chunkSize: 512,\n  overlap: 128,\n  respectBoundaries: true  // Не режем предложения\n});\n\n// Generate embeddings (OpenAI text-embedding-3-small)\nconst embeddings = await Promise.all(\n  chunks.map(c => embed(c))\n);\n\n// Store in Milvus\nawait milvus.indexDocument({\n  id: `${url}-${i}`,\n  vector: embeddings[i],\n  metadata: { url, title, timestamp, tags }\n});\n```\n\n### 2.3 Semantic Search\n\n```typescript\n// Найти похожие страницы\nconst results = await milvus.searchSimilar(query, embedding, {\n  topK: 5,\n  minScore: 0.7,\n  filter: { \n    domains: ['github.com', 'docs.python.org'],\n    dateFrom: new Date('2024-01-01')\n  }\n});\n```\n\n---\n\n## 📝 PHASE 3: Obsidian Integration\n\n### 3.1 Two-Way Sync\n\n```typescript\nclass ObsidianSyncManager {\n  // Save from browser → Obsidian\n  async saveSummary(summary: SummaryData): Promise<void> {\n    const content = this.renderTemplate(summary);\n    await this.atomicWrite(filepath, content);\n  }\n\n  // Watch for changes in Obsidian\n  startWatcher() {\n    chokidar.watch(vaultPath).on('change', (file) => {\n      const summary = this.parseNote(file);\n      this.emitChange(summary);  // Sync back to extension\n    });\n  }\n}\n```\n\n### 3.2 Note Template\n\n```markdown\n---\ncreated: 2026-02-12T10:30:00Z\nurl: https://...\ndomain: github.com\ndepth: 4\ntags: [#llm, #architecture]\ntokens: 2450\n---\n\n# {{title}}\n\n> **URL**: [{{url}}]({{url}})\n> **Domain**: {{domain}} | **Depth**: {{depth}}/5\n\n## Суть\n{{core}}\n\n## Ключевые тезисы\n{{keyPoints}}\n\n## TIL\n{{til}}\n\n## Действия\n{{actions}}\n\n## Сущности\n{{entities}}\n\n---\n*Generated by pzdrk* [[pzdrk-index]]\n```\n\n---\n\n## 🌐 PHASE 4: Local Data Hub\n\n### 4.1 REST API (localhost:7420)\n\n```typescript\n// GET /api/summaries?page=1&limit=50&domain=github.com\n// GET /api/search (semantic)\n// POST /api/summaries (create/update)\n// GET /api/stats\n// WS /ws (real-time updates)\n```\n\n### 4.2 Sync Pipeline\n\n```\nBrowser Extension\n       ↓\n   Summary Created\n       ↓\n  ┌────┴────┐\n  ↓         ↓\nMilvus   Obsidian\n(Vector)  (Files)\n  ↓         ↓\n  └────┬────┘\n       ↓\n  WebSocket Broadcast\n       ↓\n  Dashboard UI\n```\n\n---\n\n## 🚀 PHASE 5: Performance Optimizations\n\n### 5.1 Streaming & Incremental Rendering\n\n```typescript\nasync renderStreaming(stream: ReadableStream, element: HTMLElement) {\n  const reader = stream.getReader();\n  const parser = new StreamingJsonParser();\n  \n  while (true) {\n    const { done, value } = await reader.read();\n    if (done) break;\n    \n    const partial = parser.parseIncremental(value);\n    this.updateDOM(element, partial);\n    await this.frameThrottle();\n  }\n}\n```\n\n### 5.2 Predictive Prefetching\n\n```typescript\nclass PredictivePrefetchEngine {\n  async predictNextPages(currentUrl: string): Promise<Prediction[]> {\n    // ML-based prediction на основе:\n    // - Истории переходов\n    // - Ссылок на странице\n    // - Времени дня\n    return this.userModel.predictNextUrls(features, 5);\n  }\n\n  async startPrefetching() {\n    const predictions = await this.predictNextPages(location.href);\n    \n    for (const pred of predictions) {\n      scheduler.postTask(async () => {\n        const summary = await this.generateSummary(pred.url);\n        this.cache.set(pred.url, summary);\n      }, { priority: 'background' });\n    }\n  }\n}\n```\n\n---\n\n## 📦 Installation & Setup\n\n### Prerequisites\n```bash\n# macOS\nbrew install docker docker-compose node\n\n# Start Docker Desktop\nopen -a Docker\n```\n\n### Quick Start\n```bash\n# 1. Clone and setup\ncd /Users/marklindgreen/Gronxie\\ Dropbox/MARK\\ HAGGERTY/_DEV/_cli/ext\nnpm install\n\n# 2. Start Milvus\ndocker-compose -f docker-compose.milvus.yml up -d\n\n# 3. Configure keys\ncat > .env << EOF\nGROQ_API_KEYS=gsk_key1,gsk_key2,gsk_key3,gsk_key4,gsk_key5\nCEREBRAS_API_KEYS=csk_key1,csk_key2,csk_key3\nOPENAI_API_KEY=sk-...\nOBSIDIAN_VAULT_PATH=/Users/.../Documents/Obsidian/Main\nEOF\n\n# 4. Start data hub\nnpm run server\n\n# 5. Build extension\nnpm run build\n\n# 6. Load in Chrome\n# chrome://extensions → Developer mode → Load unpacked → dist/\n```\n\n---\n\n## 📊 Monitoring & Debugging\n\n### Key Metrics\n```typescript\n// GET /api/metrics\n{\n  \"providers\": {\n    \"groq\": { \"availableKeys\": 8, \"avgLatency\": 450, \"requestsPerMinute\": 45 },\n    \"cerebras\": { \"availableKeys\": 3, \"avgLatency\": 720, \"requestsPerMinute\": 12 }\n  },\n  \"cache\": { \"hitRate\": 0.34, \"size\": \"145MB\" },\n  \"milvus\": { \"totalDocuments\": 1250, \"indexSize\": \"2.3GB\" }\n}\n```\n\n### Attu GUI\nОткрыть http://localhost:8000 для просмотра:\n- Коллекций\n- Векторов\n- Query performance\n\n---\n\n## 🗓️ Roadmap\n\n| Phase | Duration | Key Deliverables |\n|-------|----------|------------------|\n| 1 | Week 1-2 | TypeScript + Vite, Smart Router |\n| 2 | Week 3-4 | Milvus integration, Key Rotation |\n| 3 | Week 5-6 | Obsidian sync, Local API |\n| 4 | Week 7-8 | Streaming, Prefetching |\n| 5 | Week 9-10 | Dashboard, CLI tools |\n| 6 | Week 11-12 | Testing, docs, release |\n\n---\n\n## 🔐 Security Considerations\n\n1. **API Keys** — хранятся в `chrome.storage.sync`, никогда не логируются\n2. **Local Milvus** — только localhost, нет external access\n3. **Obsidian Vault** — file system access через разрешения пользователя\n4. **Data retention** — пользователь контролирует все данные локально\n\n---\n\n## 💡 Future Enhancements\n\n- **Local LLM** — интеграция с Ollama/LM Studio для offline mode\n- **Graph RAG** — knowledge graph на базе страниц\n- **Collaborative** — sync между устройствами через облако\n- **Mobile app** — companion app для iOS/Android\n\n---\n\n*Generated for pzdrk v7.0 Advanced Architecture*\n"}}}}