{"repo": "brooksroley/zero-next", "files": {"docs/superpowers/specs/2026-04-28-multi-provider-arena-design.md": {"952ff5a701123ca904eb14078c2575ca25d4b9d9": {"t": 1786068575.1796496, "n": 12414, "h": "afe9bceb0ebcec75", "c": "# Multi-Provider AI Gateway & Model Arena — Design Spec\n\n**Date:** 2026-04-28\n**Status:** Draft\n**Livelihood stream:** SaaS micro-tools\n\n---\n\n## Goal\n\nUpgrade the chat sandbox from 3 OpenRouter models to ~50 free models across 3 providers (OpenRouter, Groq, Cerebras), and build a new Model Arena page at `/tools/model-arena` with side-by-side comparison and blind battle modes. All AI requests flow through a single gateway API route with auto-fallback on rate limits and optional BYOK (Bring Your Own Key) support.\n\n---\n\n## Architecture Overview\n\n```\n                      ┌──────────────┐\n                      │   Browser     │\n                      │              │\n              ┌───────┴───────┐  ┌───┴────────────┐\n              │ /tools/chat   │  │ /tools/model-   │\n              │ (sandbox)     │  │ arena           │\n              └───────┬───────┘  └───┬────────────┘\n                      │              │\n                      ▼              ▼\n              ┌──────────────────────────┐\n              │ POST /api/tools/         │\n              │ ai-gateway               │\n              │                          │\n              │ • resolve provider+model │\n              │ • resolve key (BYOK/env) │\n              │ • sanitize input         │\n              │ • stream response        │\n              │ • fallback on 429        │\n              └──────┬───┬───┬──────────┘\n                     │   │   │\n            ┌────────┘   │   └────────┐\n            ▼            ▼            ▼\n       OpenRouter      Groq       Cerebras\n       (~30 free)    (15+ free)   (5 free)\n```\n\n---\n\n## 1. Multi-Provider Registry (`src/lib/ai-providers/`)\n\n### File Structure\n\n```\nsrc/lib/ai-providers/\n  providers.ts       — provider definitions\n  models.ts          — full model registry (~50 free models)\n  fallback.ts        — cross-provider equivalent lookup + fallback selection\n  keys.ts            — API key resolution (BYOK header > server env)\n  sanitize.ts        — moved from src/lib/openrouter/sanitize.ts (unchanged)\n  colors.ts          — moved from src/lib/openrouter/colors.ts (unchanged)\n  profile-prompt.ts  — moved from src/lib/openrouter/profile-prompt.ts (unchanged)\n```\n\n### Provider Type\n\n```ts\ntype Provider = {\n  id: \"openrouter\" | \"groq\" | \"cerebras\";\n  name: string;\n  baseUrl: string;\n  envKey: string; // e.g. \"GROQ_API_KEY\"\n  rateLimit: { rpm: number; rpd: number };\n};\n```\n\nProviders defined:\n\n| id | baseUrl | envKey | Rate Limits |\n|---|---|---|---|\n| `openrouter` | `https://openrouter.ai/api/v1` | `OPENROUTER_API_KEY` | 20 RPM, 200 RPD per model |\n| `groq` | `https://api.groq.com/openai/v1` | `GROQ_API_KEY` | 30 RPM, 1000 RPD |\n| `cerebras` | `https://api.cerebras.ai/v1` | `CEREBRAS_API_KEY` | 30 RPM, 1M TPD |\n\n### Model Type\n\n```ts\ntype AIModel = {\n  id: string;               // unique, e.g. \"groq/llama-3.3-70b\"\n  providerId: string;\n  providerModelId: string;  // what provider expects, e.g. \"llama-3.3-70b-versatile\"\n  displayName: string;\n  contextLength: number;\n  capabilities: (\"chat\" | \"vision\" | \"code\")[];\n  free: boolean;\n  equivalentIds?: string[]; // cross-provider duplicates for fallback\n};\n```\n\n### Model Roster (Tier 1 — ~50 models)\n\n**OpenRouter (~30 free models):**\n- NVIDIA Nemotron 3 Super (120B, 262K context)\n- OpenAI GPT-OSS 120B / 20B\n- Meta Llama 3.3 70B / Llama 4 Scout / Llama 4 Maverick\n- Qwen3 Coder 480B / Qwen3 235B Thinking\n- Google Gemma 4 31B / Gemma 3 27B / 12B / 4B\n- Mistral Small 3.1 24B / Devstral 2 123B\n- MiniMax M2.5\n- Z.AI GLM 4.5 Air\n- Arcee AI Trinity Large Preview\n- Nous Hermes 3 405B\n- inclusionAI Ling-2.6-flash\n- Gemma 3n 4B / 2B\n- OpenRouter Free Models Router (auto-select)\n\n**Groq (15+ free models):**\n- Llama 4 Scout (512K) / Llama 4 Maverick (256K)\n- Llama 3.3 70B Versatile / Llama 3.1 8B Instant\n- Qwen QwQ-32B\n- GPT-OSS 120B / 20B\n- DeepSeek R1 Distill 70B\n- Mistral Saba 24B\n- Gemma 2 9B IT\n\n**Cerebras (5 free models):**\n- Qwen3 235B A22B Instruct (64K, ~1400 tok/s)\n- GPT-OSS 120B (~3000 tok/s)\n- Qwen3 Coder 480B\n- Llama 3.1 8B (~1800 tok/s)\n- Z.AI GLM-4.7\n\n### Fallback / Cross-Listing Map\n\nModels that exist on multiple providers share `equivalentIds`. When provider A returns 429, the gateway tries provider B for the same model.\n\nCross-listed models:\n- Llama 3.3 70B → OpenRouter, Groq\n- Llama 4 Scout → OpenRouter, Groq\n- Llama 4 Maverick → OpenRouter, Groq\n- GPT-OSS 120B → OpenRouter, Groq, Cerebras\n- GPT-OSS 20B → OpenRouter, Groq\n- Llama 3.1 8B → Groq, Cerebras\n- Qwen3 235B → OpenRouter, Cerebras\n- Qwen3 Coder 480B → OpenRouter, Cerebras\n- Mistral Small 3.1 → OpenRouter\n- GLM 4.5/4.7 → OpenRouter, Cerebras\n\n### Key Resolution (`keys.ts`)\n\n```ts\nfunction resolveKey(providerId: string, byokHeader?: string): string | null\n```\n\n1. If `X-Provider-Key` header is present and non-empty, use it\n2. Else read `process.env[provider.envKey]`\n3. If neither exists, return null (gateway returns 401)\n\n---\n\n## 2. AI Gateway API Route\n\n**Path:** `POST /api/tools/ai-gateway`\n\n### Request\n\n```ts\n{\n  modelId: string;          // from registry, e.g. \"groq/llama-3.3-70b\"\n  systemPrompt?: string;    // for character chat\n  messages: { role: \"user\" | \"assistant\"; content: string }[];\n  maxTokens?: number;       // default 2000\n  temperature?: number;     // default 0.8\n}\n\nHeaders:\n  X-Provider-Key?: string   // optional BYOK\n```\n\n### Response\n\nStreaming text via `pipeTextStreamToResponse`. On error:\n- 400: bad request (missing fields, unknown model, sanitization failure)\n- 401: no API key available for this provider\n- 429: all providers exhausted for this model\n- 502: upstream provider error\n\n### Flow\n\n1. Look up `modelId` in registry → get `AIModel` + `Provider`\n2. Resolve API key via `keys.ts`\n3. Sanitize last user message via `sanitize.ts`\n4. Create SDK client using `@ai-sdk/openai-compatible` with provider's `baseUrl` + resolved key\n5. Call `streamText()` and pipe to response\n6. On 429 error: look up `equivalentIds`, try next provider that has a key\n7. On all-providers-exhausted: return 429\n\n### SDK Client Creation\n\nAll three providers are OpenAI-compatible. Use `@ai-sdk/openai-compatible`:\n\n```ts\nimport { createOpenAICompatible } from \"@ai-sdk/openai-compatible\";\n\nfunction createProviderClient(provider: Provider, apiKey: string) {\n  return createOpenAICompatible({\n    name: provider.id,\n    baseURL: provider.baseUrl,\n    apiKey,\n  });\n}\n```\n\nOpenRouter stays on `@openrouter/ai-sdk-provider` (already installed) since it has specific features. Groq and Cerebras use the generic compatible client.\n\n---\n\n## 3. Chat Sandbox Upgrade\n\n### Changes to `src/pages/tools/chat.tsx`\n\n- **Model dropdown**: Replace flat list with optgroup-by-provider. Shows provider name as group header, models within. ~50 options total.\n- **API endpoint**: `streamCharacterResponse` calls `/api/tools/ai-gateway` instead of `/api/tools/chat`\n- **Provider badge**: Character strip shows small \"via Groq\" / \"via Cerebras\" text under model name\n- **BYOK panel**: Collapsible \"API Keys\" section in the header. Three fields (OpenRouter, Groq, Cerebras). Values saved to localStorage under `ai-provider-keys`. Sent as `X-Provider-Key` header on requests matching that provider.\n\n### Changes to `src/pages/api/tools/generate-profile.ts`\n\n- Use the gateway's provider resolution internally instead of hardcoded `createOpenRouter`\n- Default profile model stays the first OpenRouter model (or configurable)\n\n### No Changes\n\n- Character CRUD (`characters.ts`) — unchanged\n- `characters` table schema — unchanged (model column stores string IDs)\n- Conversation state management — unchanged\n- UI layout structure — unchanged\n\n---\n\n## 4. Model Arena Page\n\n**Path:** `/tools/model-arena`\n\n### Layout\n\n```\n┌─────────────────────────────────────┐\n│ ← Home    Model Arena    [API Keys] │\n│─────────────────────────────────────│\n│  [Compare]  [Battle]                │\n│─────────────────────────────────────│\n│                                     │\n│   ┌─────────────┐ ┌─────────────┐  │\n│   │  Model A     │ │  Model B     │ │\n│   │  (streaming) │ │  (streaming) │ │\n│   │              │ │              │ │\n│   │  1.2s TTFT   │ │  0.3s TTFT   │ │\n│   │  847 tokens  │ │  923 tokens  │ │\n│   └─────────────┘ └─────────────┘  │\n│                                     │\n│─────────────────────────────────────│\n│ [prompt input]              [Send]  │\n└─────────────────────────────────────┘\n```\n\n### Mode 1: Side-by-Side Comparison\n\n- User picks 2 or 3 models from dropdown selectors above each column\n- Types prompt, hits Send\n- Fires parallel requests to `/api/tools/ai-gateway` via `Promise.all`\n- Each column streams independently, showing:\n  - Model name + provider badge\n  - Streaming response text\n  - Time-to-first-token (measured client-side: time from fetch to first chunk)\n  - Total response time\n  - Approximate token count (word count / 0.75)\n- \"Re-run\" button replays same prompt, \"Clear\" resets\n\n### Mode 2: Blind Battle\n\n- User types prompt, hits Send\n- System randomly picks 2 models from the free pool, weighted to prefer different providers\n- Responses stream in columns labeled \"Model A\" / \"Model B\" — no names\n- After both complete, two buttons appear: \"A wins\" / \"B wins\" / \"Tie\"\n- On vote: models revealed with names, provider, and speed stats\n- Battle history stored in localStorage:\n  ```ts\n  type BattleRecord = {\n    modelA: string;\n    modelB: string;\n    winner: \"a\" | \"b\" | \"tie\";\n    prompt: string;\n    timestamp: number;\n  };\n  ```\n- Running leaderboard shown in a collapsible sidebar: model win rates from local battles\n\n### Shared Components\n\n- BYOK key panel — same component as chat sandbox, shared localStorage\n- Model selector dropdown — shared component from `src/lib/ai-providers/`\n- Streaming display column — reusable component with TTFT/speed metrics\n\n---\n\n## 5. BYOK (Bring Your Own Key) UX\n\n### Storage\n\nlocalStorage key: `ai-provider-keys`\n\n```ts\ntype StoredKeys = {\n  openrouter?: string;\n  groq?: string;\n  cerebras?: string;\n};\n```\n\n### UI\n\nCollapsible panel accessible from both chat sandbox and model arena headers. Shows:\n- One input per provider with obscured display (type=\"password\")\n- \"Save\" persists to localStorage\n- \"Clear\" removes a key\n- Green dot indicator next to provider name when key is saved\n- Disclaimer: \"Keys are stored in your browser only and sent directly to the provider. We never log or store them.\"\n\n### Request Flow\n\nWhen a BYOK key exists for the target provider, the client sends it in the `X-Provider-Key` request header. The gateway uses it instead of the server env var. If the BYOK key is invalid (401 from upstream), the error surfaces to the user — no fallback to server key (that would leak the user's intent to use their own key).\n\n---\n\n## 6. Cleanup & Migration\n\n### Delete\n- `src/pages/api/tools/setup-characters.ts` — one-time migration, table exists\n- `src/pages/api/tools/chat.ts` — replaced by ai-gateway\n\n### Move (content unchanged, new paths)\n- `src/lib/openrouter/sanitize.ts` → `src/lib/ai-providers/sanitize.ts`\n- `src/lib/openrouter/colors.ts` → `src/lib/ai-providers/colors.ts`\n- `src/lib/openrouter/profile-prompt.ts` → `src/lib/ai-providers/profile-prompt.ts`\n\n### Replace\n- `src/lib/openrouter/models.ts` → replaced by `src/lib/ai-providers/models.ts` (new registry)\n\n### Delete after move\n- `src/lib/openrouter/` directory (empty after moves)\n\n### Update Imports\n- `src/pages/tools/chat.tsx` — import from `src/lib/ai-providers/`\n- `src/pages/api/tools/generate-profile.ts` — use new provider resolution\n- `src/pages/api/tools/characters.ts` — no changes needed\n\n### New Dependencies\n- `@ai-sdk/openai-compatible` — generic OpenAI-compatible provider for Groq + Cerebras\n\n### New Env Vars\n- `GROQ_API_KEY` — free at groq.com, no credit card\n- `CEREBRAS_API_KEY` — free at cerebras.ai, no credit card\n- `OPENROUTER_API_KEY` — already exists\n\n### Homepage\n- Add Model Arena card to `src/pages/index.tsx` alongside existing Chat Sandbox card\n\n### No Database Changes\n- `characters` table unchanged\n- Arena battle tallies are localStorage only\n\n---\n\n## Non-Goals (explicitly deferred)\n\n- Tier 2 providers (Mistral, Google Gemini) — add later when Tier 1 is stable\n- Server-side battle leaderboard / database persistence — localStorage first\n- Paid model support via BYOK — free models only for now\n- Chat history persistence — conversations remain ephemeral React state\n- Model speed benchmarking database — client-side timing only\n"}}}}