1 line
39 KiB
JSON
1 line
39 KiB
JSON
{"repo": "actuallypretending/howfastwould", "files": {"docs/superpowers/plans/2026-04-07-free-providers.md": {"b80099dc4561a155d0b20eb444bc12eb1c5eff06": {"t": 1785746457.0528743, "n": 19063, "h": "b9fc65f92f0f1425", "c": "# Free Providers Implementation Plan\n\n> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.\n\n**Goal:** Add Groq, GitHub Models, and Cloudflare Workers AI providers to the benchmark leaderboard alongside existing free-tier models (Gemini, DeepSeek, Qwen already seeded), with an on-demand trigger that benchmarks missing models when a user views a problem.\n\n**Architecture:** New provider branches in `runner.rs` handle API format differences. `sync.rs` seed is changed to always upsert (removing the \"skip if models exist\" guard) so new models appear on existing deployments. `routes/problems.rs` spawns a background Tokio task when active models have no result for the requested problem.\n\n**Tech Stack:** Rust, axum, sqlx (PostgreSQL), tokio, reqwest\n\n---\n\n## File Structure\n\n| File | Change |\n|------|--------|\n| `backend/src/runner.rs` | Add `groq`, `github`, `cloudflare` to `build_api_request`; add `cloudflare` to `extract_code` |\n| `backend/src/config.rs` | Add `groq_api_key`, `github_token`, `cf_api_token`, `cf_account_id` fields |\n| `backend/src/sync.rs` | Remove early-return count guard; change seed INSERT to `ON CONFLICT (name) DO NOTHING`; add 5 new model rows |\n| `backend/src/routes/problems.rs` | Add on-demand trigger: find missing models, spawn background benchmark |\n| `backend/.env.example` | Document 4 new env vars |\n\n---\n\n### Task 1: Add groq, github, cloudflare provider support to runner.rs\n\n**Files:**\n- Modify: `backend/src/runner.rs`\n\n**Context:** `build_api_request` is a pure match function returning `(url, body)`. Groq and GitHub Models are OpenAI-compatible (same body format, different base URL). Cloudflare uses a different URL structure (`/accounts/{CF_ACCOUNT_ID}/ai/run/{model}`) and a different response shape (`result.response` instead of `choices[0].message.content`). Auth headers are handled separately by `auth_header_name`/`auth_header_value` and work correctly for all three (all use `Authorization: Bearer <key>`).\n\n- [ ] **Step 1: Write failing tests**\n\nAdd at the bottom of `backend/src/runner.rs`:\n\n```rust\n#[cfg(test)]\nmod tests {\n use super::*;\n\n #[test]\n fn test_build_api_request_groq() {\n let (url, body) = build_api_request(\"groq\", \"llama-3.3-70b-versatile\", \"\", \"test\").unwrap();\n assert_eq!(url, \"https://api.groq.com/openai/v1/chat/completions\");\n assert_eq!(body[\"model\"], \"llama-3.3-70b-versatile\");\n assert!(body[\"messages\"].is_array());\n }\n\n #[test]\n fn test_build_api_request_github() {\n let (url, body) = build_api_request(\"github\", \"gpt-4o-mini\", \"\", \"test\").unwrap();\n assert_eq!(url, \"https://models.inference.ai.azure.com/chat/completions\");\n assert_eq!(body[\"model\"], \"gpt-4o-mini\");\n }\n\n #[test]\n fn test_build_api_request_cloudflare() {\n std::env::set_var(\"CF_ACCOUNT_ID\", \"abc123\");\n let (url, _) = build_api_request(\"cloudflare\", \"@cf/meta/llama-3.1-8b-instruct\", \"\", \"test\").unwrap();\n assert!(url.contains(\"abc123\"), \"URL should contain account ID\");\n assert!(url.contains(\"%40cf\") || url.contains(\"@cf\"), \"URL should contain model name\");\n }\n\n #[test]\n fn test_build_api_request_unknown_provider_errors() {\n let result = build_api_request(\"notreal\", \"model\", \"\", \"test\");\n assert!(result.is_err());\n }\n\n #[test]\n fn test_extract_code_cloudflare() {\n let resp = serde_json::json!({\n \"result\": { \"response\": \"def solution():\\n return 42\" },\n \"success\": true\n });\n let code = extract_code(&resp, \"cloudflare\").unwrap();\n assert_eq!(code, \"def solution():\\n return 42\");\n }\n\n #[test]\n fn test_extract_code_cloudflare_with_code_fence() {\n let resp = serde_json::json!({\n \"result\": { \"response\": \"```python\\ndef solution():\\n return 42\\n```\" }\n });\n let code = extract_code(&resp, \"cloudflare\").unwrap();\n assert_eq!(code, \"def solution():\\n return 42\");\n }\n\n #[test]\n fn test_extract_code_cloudflare_missing_field() {\n let resp = serde_json::json!({ \"result\": {} });\n let code = extract_code(&resp, \"cloudflare\").unwrap();\n assert_eq!(code, \"\");\n }\n}\n```\n\n- [ ] **Step 2: Run tests to verify they fail**\n\n```bash\ncd /path/to/howfastwould\ncargo test --manifest-path backend/Cargo.toml 2>&1 | tail -20\n```\n\nExpected: compile error or test failures mentioning \"groq\", \"github\", \"cloudflare\" not matched.\n\n- [ ] **Step 3: Add groq and github to `build_api_request`**\n\nIn `backend/src/runner.rs`, find the `build_api_request` function. The first match arm is:\n```rust\n\"openai\" | \"xai\" | \"fireworks\" | \"deepseek\" | \"mistral\" => {\n```\n\nChange it to include `groq` and `github` in the same arm (they use the same OpenAI-compatible format):\n\n```rust\n\"openai\" | \"xai\" | \"fireworks\" | \"deepseek\" | \"mistral\" | \"groq\" | \"github\" => {\n let base = match provider {\n \"openai\" => \"https://api.openai.com/v1\",\n \"xai\" => \"https://api.x.ai/v1\",\n \"fireworks\" => \"https://api.fireworks.ai/inference/v1\",\n \"deepseek\" => \"https://api.deepseek.com/v1\",\n \"mistral\" => \"https://api.mistral.ai/v1\",\n \"groq\" => \"https://api.groq.com/openai/v1\",\n \"github\" => \"https://models.inference.ai.azure.com\",\n _ => unreachable!(),\n };\n Ok((\n format!(\"{}/chat/completions\", base),\n json!({ \"model\": model_name, \"messages\": [{\"role\":\"user\",\"content\": prompt}], \"max_tokens\": 2048 })\n ))\n}\n```\n\n- [ ] **Step 4: Add cloudflare to `build_api_request`**\n\nAfter the `\"google\" => ...` arm (line ~206), add:\n\n```rust\n\"cloudflare\" => {\n let account_id = std::env::var(\"CF_ACCOUNT_ID\").unwrap_or_default();\n Ok((\n format!(\n \"https://api.cloudflare.com/client/v4/accounts/{}/ai/run/{}\",\n account_id, model_name\n ),\n json!({ \"messages\": [{\"role\":\"user\",\"content\": prompt}], \"max_tokens\": 2048 })\n ))\n},\n```\n\n- [ ] **Step 5: Add cloudflare to `extract_code`**\n\nIn `extract_code`, add a `\"cloudflare\"` arm before the default `_` arm:\n\n```rust\nfn extract_code(resp: &Value, provider: &str) -> Result<String> {\n let text = match provider {\n \"anthropic\" => resp[\"content\"][0][\"text\"].as_str().unwrap_or(\"\"),\n \"google\" => resp[\"candidates\"][0][\"content\"][\"parts\"][0][\"text\"].as_str().unwrap_or(\"\"),\n \"qwen\" => resp[\"output\"][\"text\"].as_str().unwrap_or(\"\"),\n \"hunyuan\" => resp[\"Choices\"][0][\"Message\"][\"Content\"].as_str().unwrap_or(\"\"),\n \"cloudflare\" => resp[\"result\"][\"response\"].as_str().unwrap_or(\"\"),\n _ => resp[\"choices\"][0][\"message\"][\"content\"].as_str().unwrap_or(\"\"),\n };\n // ... rest of function unchanged\n```\n\n- [ ] **Step 6: Run tests to verify they pass**\n\n```bash\ncargo test --manifest-path backend/Cargo.toml 2>&1 | tail -20\n```\n\nExpected: all new tests pass. Note: `test_build_api_request_cloudflare` sets `CF_ACCOUNT_ID` env var — if tests run in parallel this is fine since each test sets it before asserting.\n\n- [ ] **Step 7: Commit**\n\n```bash\ngit add backend/src/runner.rs\ngit commit -m \"feat: add groq, github, cloudflare provider support\"\n```\n\n---\n\n### Task 2: Add config fields for new providers\n\n**Files:**\n- Modify: `backend/src/config.rs`\n\n**Context:** `config.rs` holds all API keys loaded from env vars on startup. The `runner.rs` reads API keys at race time via `std::env::var(&model.api_key_env)`, so these config fields aren't strictly required for the runner — but they're needed for the `sync_models` auto-discovery function in `sync.rs` (which checks `if !config.groq_api_key.is_empty()`). Add them now so `sync.rs` can be extended in the future. `CF_ACCOUNT_ID` is used directly in `build_api_request` via `std::env::var`, so it doesn't need a config field — but add it for completeness and testability.\n\n- [ ] **Step 1: Write failing test**\n\nAdd to `backend/src/config.rs`:\n\n```rust\n#[cfg(test)]\nmod tests {\n use super::*;\n\n #[test]\n fn test_config_new_fields_default_empty() {\n // Ensure new fields don't panic when env vars are absent\n std::env::remove_var(\"GROQ_API_KEY\");\n std::env::remove_var(\"GITHUB_TOKEN\");\n std::env::remove_var(\"CF_API_TOKEN\");\n std::env::remove_var(\"CF_ACCOUNT_ID\");\n let cfg = Config::from_env().unwrap();\n assert_eq!(cfg.groq_api_key, \"\");\n assert_eq!(cfg.github_token, \"\");\n assert_eq!(cfg.cf_api_token, \"\");\n assert_eq!(cfg.cf_account_id, \"\");\n }\n}\n```\n\n- [ ] **Step 2: Run test to verify it fails**\n\n```bash\ncargo test --manifest-path backend/Cargo.toml config::tests 2>&1\n```\n\nExpected: compile error — fields don't exist yet.\n\n- [ ] **Step 3: Add fields to Config struct**\n\nIn `backend/src/config.rs`, add to the `Config` struct after `mistral_api_key`:\n\n```rust\npub groq_api_key: String,\npub github_token: String,\npub cf_api_token: String,\npub cf_account_id: String,\n```\n\n- [ ] **Step 4: Add fields to `from_env`**\n\nIn `from_env()`, add after `mistral_api_key: std::env::var(\"MISTRAL_API_KEY\").unwrap_or_default(),`:\n\n```rust\ngroq_api_key: std::env::var(\"GROQ_API_KEY\").unwrap_or_default(),\ngithub_token: std::env::var(\"GITHUB_TOKEN\").unwrap_or_default(),\ncf_api_token: std::env::var(\"CF_API_TOKEN\").unwrap_or_default(),\ncf_account_id: std::env::var(\"CF_ACCOUNT_ID\").unwrap_or_default(),\n```\n\n- [ ] **Step 5: Run test to verify it passes**\n\n```bash\ncargo test --manifest-path backend/Cargo.toml config::tests 2>&1\n```\n\nExpected: PASS.\n\n- [ ] **Step 6: Commit**\n\n```bash\ngit add backend/src/config.rs\ngit commit -m \"feat: add groq, github, cloudflare fields to Config\"\n```\n\n---\n\n### Task 3: Fix seed and add new model rows\n\n**Files:**\n- Modify: `backend/src/sync.rs`\n\n**Context:** `seed_initial_models` currently short-circuits with `if count > 0 { return Ok(()); }` and uses a plain INSERT with no conflict handling. This means new model rows never get added to existing deployments. Fix: remove the count guard and add `ON CONFLICT (name) DO NOTHING` to the INSERT so it's safe to run on every startup. Then add the 5 new model rows (groq ×2, github ×2, cloudflare ×1). Note: gemini-2.0-flash, deepseek-chat, deepseek-reasoner, and qwen models are already in the seed list.\n\n- [ ] **Step 1: Remove the count guard**\n\nIn `backend/src/sync.rs`, find `seed_initial_models`. Remove these lines:\n\n```rust\nlet count: i64 = sqlx::query_scalar!(\"SELECT COUNT(*) FROM models\")\n .fetch_one(pool).await?.unwrap_or(0);\n\nif count > 0 { return Ok(()); }\n```\n\n- [ ] **Step 2: Add `ON CONFLICT (name) DO NOTHING` to the seed INSERT**\n\nFind the `sqlx::query!` call inside the `for` loop in `seed_initial_models`. Change:\n\n```rust\nsqlx::query!(\n \"INSERT INTO models (id, provider, name, display_name, api_key_env, is_active, is_new, is_human, human_times, added_at) VALUES ($1, $2, $3, $4, $5, true, false, $6, $7, $8)\",\n id, provider, name, display, key_env, is_human, human_times, now\n).execute(pool).await?;\n```\n\nTo:\n\n```rust\nsqlx::query!(\n \"INSERT INTO models (id, provider, name, display_name, api_key_env, is_active, is_new, is_human, human_times, added_at) VALUES ($1, $2, $3, $4, $5, true, false, $6, $7, $8) ON CONFLICT (name) DO NOTHING\",\n id, provider, name, display, key_env, is_human, human_times, now\n).execute(pool).await?;\n```\n\n- [ ] **Step 3: Add new model rows to the seed list**\n\nFind the `models: &[(&str, &str, &str, &str, bool)]` slice in `seed_initial_models`. Add these 5 rows after the existing entries and before the human rows:\n\n```rust\n(\"groq\", \"llama-3.3-70b-versatile\", \"Llama 3.3 70B\", \"GROQ_API_KEY\", false),\n(\"groq\", \"mixtral-8x7b-32768\", \"Mixtral 8x7B\", \"GROQ_API_KEY\", false),\n(\"github\", \"gpt-4o-mini\", \"GPT-4o mini\", \"GITHUB_TOKEN\", false),\n(\"github\", \"Meta-Llama-3-70B-Instruct\", \"Llama 3 70B\", \"GITHUB_TOKEN\", false),\n(\"cloudflare\", \"@cf/meta/llama-3.1-8b-instruct\", \"Llama 3.1 8B\", \"CF_API_TOKEN\", false),\n```\n\n- [ ] **Step 4: Verify compilation**\n\n```bash\ncargo check --manifest-path backend/Cargo.toml 2>&1\n```\n\nExpected: no errors. (sqlx compile-time query checks run against `DATABASE_URL` — if not set locally, use `SQLX_OFFLINE=true cargo check ...`)\n\n- [ ] **Step 5: Commit**\n\n```bash\ngit add backend/src/sync.rs\ngit commit -m \"feat: seed groq, github, cloudflare models; always upsert on startup\"\n```\n\n---\n\n### Task 4: On-demand benchmark trigger in problems.rs\n\n**Files:**\n- Modify: `backend/src/routes/problems.rs`\n\n**Context:** The `results` handler currently fetches and returns race results for a problem. We extend it to: (1) check which active non-human models have no result for this problem, (2) if any are missing, spawn a `tokio::spawn` background task that calls `runner.race()` and saves results. The HTTP response returns immediately with existing results — the frontend's polling loop surfaces new results as they complete. `AppState` already has `runner: Arc<Runner>` and `pool: PgPool`.\n\n- [ ] **Step 1: Write a test for the missing-model detection logic**\n\nThe detection logic (finding model IDs with no results) is pure set math. Write a unit test for it:\n\n```rust\n#[cfg(test)]\nmod tests {\n use std::collections::HashSet;\n\n #[test]\n fn test_missing_model_ids() {\n let benchmarked: HashSet<String> = [\"model-a\", \"model-b\"]\n .iter().map(|s| s.to_string()).collect();\n let all_active = vec![\"model-a\", \"model-b\", \"model-c\", \"model-d\"];\n let missing: Vec<&str> = all_active.into_iter()\n .filter(|id| !benchmarked.contains(*id))\n .collect();\n assert_eq!(missing, vec![\"model-c\", \"model-d\"]);\n }\n\n #[test]\n fn test_no_missing_models_when_all_benchmarked() {\n let benchmarked: HashSet<String> = [\"model-a\"].iter().map(|s| s.to_string()).collect();\n let all_active = vec![\"model-a\"];\n let missing: Vec<&str> = all_active.into_iter()\n .filter(|id| !benchmarked.contains(*id))\n .collect();\n assert!(missing.is_empty());\n }\n}\n```\n\nAdd this at the bottom of `backend/src/routes/problems.rs`.\n\n- [ ] **Step 2: Run tests to verify they pass (pure logic, no DB needed)**\n\n```bash\ncargo test --manifest-path backend/Cargo.toml routes::problems 2>&1\n```\n\nExpected: 2 tests pass.\n\n- [ ] **Step 3: Add imports to problems.rs**\n\nAt the top of `backend/src/routes/problems.rs`, add to the existing `use` block:\n\n```rust\nuse std::collections::HashSet;\nuse tokio::sync::broadcast;\nuse uuid::Uuid;\nuse crate::models::Model;\n```\n\n- [ ] **Step 4: Replace the `results` handler with the on-demand trigger version**\n\nReplace the entire `results` function in `backend/src/routes/problems.rs`:\n\n```rust\npub async fn results(\n State(state): State<AppState>,\n Path(id): Path<String>,\n) -> Json<Vec<RaceResultWithModel>> {\n let rows = sqlx::query!(\n r#\"SELECT r.id, r.problem_id, r.model_id, r.solved, r.time_ms, r.attempts, r.run_at,\n m.display_name, m.provider, m.name as model_name, m.is_human\n FROM results r JOIN models m ON r.model_id = m.id\n WHERE r.problem_id = $1\n ORDER BY r.time_ms ASC NULLS LAST\"#,\n id\n ).fetch_all(&state.pool).await.unwrap_or_default();\n\n let results: Vec<RaceResultWithModel> = rows.into_iter().map(|r| RaceResultWithModel {\n model_id: r.model_id,\n model_name: r.model_name,\n display_name: r.display_name,\n provider: r.provider,\n is_human: r.is_human,\n solved: r.solved,\n time_ms: r.time_ms,\n attempts: r.attempts,\n run_at: r.run_at,\n }).collect();\n\n // On-demand trigger: find active models with no result for this problem\n let benchmarked_ids: HashSet<String> = results.iter()\n .filter(|r| !r.is_human)\n .map(|r| r.model_id.clone())\n .collect();\n\n let all_active = sqlx::query_as!(\n Model,\n \"SELECT * FROM models WHERE is_active = true AND is_human = false\"\n ).fetch_all(&state.pool).await.unwrap_or_default();\n\n let missing: Vec<Model> = all_active.into_iter()\n .filter(|m| !benchmarked_ids.contains(&m.id))\n .collect();\n\n if !missing.is_empty() {\n let problem_result = sqlx::query_as!(\n crate::models::Problem,\n \"SELECT * FROM problems WHERE id = $1\",\n id\n ).fetch_optional(&state.pool).await.unwrap_or(None);\n\n if let Some(problem) = problem_result {\n let runner = state.runner.clone();\n let pool = state.pool.clone();\n let race_id = Uuid::new_v4().to_string();\n let (tx, _) = broadcast::channel(64);\n tokio::spawn(async move {\n tracing::info!(\n \"on-demand benchmark: {} missing models for problem {}\",\n missing.len(), problem.id\n );\n let results = runner.race(&race_id, &problem, missing, tx).await;\n for result in &results {\n sqlx::query!(\n r#\"INSERT INTO results (id, problem_id, model_id, solved, time_ms, attempts, run_at)\n VALUES ($1, $2, $3, $4, $5, $6, $7)\n ON CONFLICT (id) DO NOTHING\"#,\n result.id, result.problem_id, result.model_id,\n result.solved, result.time_ms, result.attempts, result.run_at\n ).execute(&pool).await.ok();\n }\n });\n }\n }\n\n Json(results)\n}\n```\n\n- [ ] **Step 5: Verify compilation**\n\n```bash\ncargo check --manifest-path backend/Cargo.toml 2>&1\n```\n\nExpected: no errors.\n\n- [ ] **Step 6: Commit**\n\n```bash\ngit add backend/src/routes/problems.rs\ngit commit -m \"feat: on-demand benchmark trigger when models are missing results\"\n```\n\n---\n\n### Task 5: Document new env vars\n\n**Files:**\n- Modify: `backend/.env.example`\n\n- [ ] **Step 1: Add new env vars to .env.example**\n\nOpen `backend/.env.example` and add after the existing key entries:\n\n```\n# Free-tier providers\nGROQ_API_KEY=\nGITHUB_TOKEN=\nCF_API_TOKEN=\nCF_ACCOUNT_ID=\n```\n\n- [ ] **Step 2: Verify no other changes needed**\n\n```bash\ncargo check --manifest-path backend/Cargo.toml 2>&1\n```\n\nExpected: clean.\n\n- [ ] **Step 3: Commit**\n\n```bash\ngit add backend/.env.example\ngit commit -m \"docs: add free provider env vars to .env.example\"\n```\n\n---\n\n## Post-implementation\n\nAfter all tasks are complete, set env vars in Railway for any providers you have keys for:\n\n```\nGROQ_API_KEY=<from console.groq.com>\nGITHUB_TOKEN=<GitHub personal access token with Models permission>\nCF_API_TOKEN=<from Cloudflare dashboard>\nCF_ACCOUNT_ID=<from Cloudflare dashboard>\n```\n\nOn next deploy, `seed_initial_models` will insert the 5 new model rows. On first user visit to any problem, the on-demand trigger fires and benchmarks all missing models in the background.\n"}, "4c7f2e72847a881df5705045b03ab31fbf84818d": {"t": 1785764524.6509578, "n": 19063, "h": "b9fc65f92f0f1425", "c": "# Free Providers Implementation Plan\n\n> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.\n\n**Goal:** Add Groq, GitHub Models, and Cloudflare Workers AI providers to the benchmark leaderboard alongside existing free-tier models (Gemini, DeepSeek, Qwen already seeded), with an on-demand trigger that benchmarks missing models when a user views a problem.\n\n**Architecture:** New provider branches in `runner.rs` handle API format differences. `sync.rs` seed is changed to always upsert (removing the \"skip if models exist\" guard) so new models appear on existing deployments. `routes/problems.rs` spawns a background Tokio task when active models have no result for the requested problem.\n\n**Tech Stack:** Rust, axum, sqlx (PostgreSQL), tokio, reqwest\n\n---\n\n## File Structure\n\n| File | Change |\n|------|--------|\n| `backend/src/runner.rs` | Add `groq`, `github`, `cloudflare` to `build_api_request`; add `cloudflare` to `extract_code` |\n| `backend/src/config.rs` | Add `groq_api_key`, `github_token`, `cf_api_token`, `cf_account_id` fields |\n| `backend/src/sync.rs` | Remove early-return count guard; change seed INSERT to `ON CONFLICT (name) DO NOTHING`; add 5 new model rows |\n| `backend/src/routes/problems.rs` | Add on-demand trigger: find missing models, spawn background benchmark |\n| `backend/.env.example` | Document 4 new env vars |\n\n---\n\n### Task 1: Add groq, github, cloudflare provider support to runner.rs\n\n**Files:**\n- Modify: `backend/src/runner.rs`\n\n**Context:** `build_api_request` is a pure match function returning `(url, body)`. Groq and GitHub Models are OpenAI-compatible (same body format, different base URL). Cloudflare uses a different URL structure (`/accounts/{CF_ACCOUNT_ID}/ai/run/{model}`) and a different response shape (`result.response` instead of `choices[0].message.content`). Auth headers are handled separately by `auth_header_name`/`auth_header_value` and work correctly for all three (all use `Authorization: Bearer <key>`).\n\n- [ ] **Step 1: Write failing tests**\n\nAdd at the bottom of `backend/src/runner.rs`:\n\n```rust\n#[cfg(test)]\nmod tests {\n use super::*;\n\n #[test]\n fn test_build_api_request_groq() {\n let (url, body) = build_api_request(\"groq\", \"llama-3.3-70b-versatile\", \"\", \"test\").unwrap();\n assert_eq!(url, \"https://api.groq.com/openai/v1/chat/completions\");\n assert_eq!(body[\"model\"], \"llama-3.3-70b-versatile\");\n assert!(body[\"messages\"].is_array());\n }\n\n #[test]\n fn test_build_api_request_github() {\n let (url, body) = build_api_request(\"github\", \"gpt-4o-mini\", \"\", \"test\").unwrap();\n assert_eq!(url, \"https://models.inference.ai.azure.com/chat/completions\");\n assert_eq!(body[\"model\"], \"gpt-4o-mini\");\n }\n\n #[test]\n fn test_build_api_request_cloudflare() {\n std::env::set_var(\"CF_ACCOUNT_ID\", \"abc123\");\n let (url, _) = build_api_request(\"cloudflare\", \"@cf/meta/llama-3.1-8b-instruct\", \"\", \"test\").unwrap();\n assert!(url.contains(\"abc123\"), \"URL should contain account ID\");\n assert!(url.contains(\"%40cf\") || url.contains(\"@cf\"), \"URL should contain model name\");\n }\n\n #[test]\n fn test_build_api_request_unknown_provider_errors() {\n let result = build_api_request(\"notreal\", \"model\", \"\", \"test\");\n assert!(result.is_err());\n }\n\n #[test]\n fn test_extract_code_cloudflare() {\n let resp = serde_json::json!({\n \"result\": { \"response\": \"def solution():\\n return 42\" },\n \"success\": true\n });\n let code = extract_code(&resp, \"cloudflare\").unwrap();\n assert_eq!(code, \"def solution():\\n return 42\");\n }\n\n #[test]\n fn test_extract_code_cloudflare_with_code_fence() {\n let resp = serde_json::json!({\n \"result\": { \"response\": \"```python\\ndef solution():\\n return 42\\n```\" }\n });\n let code = extract_code(&resp, \"cloudflare\").unwrap();\n assert_eq!(code, \"def solution():\\n return 42\");\n }\n\n #[test]\n fn test_extract_code_cloudflare_missing_field() {\n let resp = serde_json::json!({ \"result\": {} });\n let code = extract_code(&resp, \"cloudflare\").unwrap();\n assert_eq!(code, \"\");\n }\n}\n```\n\n- [ ] **Step 2: Run tests to verify they fail**\n\n```bash\ncd /path/to/howfastwould\ncargo test --manifest-path backend/Cargo.toml 2>&1 | tail -20\n```\n\nExpected: compile error or test failures mentioning \"groq\", \"github\", \"cloudflare\" not matched.\n\n- [ ] **Step 3: Add groq and github to `build_api_request`**\n\nIn `backend/src/runner.rs`, find the `build_api_request` function. The first match arm is:\n```rust\n\"openai\" | \"xai\" | \"fireworks\" | \"deepseek\" | \"mistral\" => {\n```\n\nChange it to include `groq` and `github` in the same arm (they use the same OpenAI-compatible format):\n\n```rust\n\"openai\" | \"xai\" | \"fireworks\" | \"deepseek\" | \"mistral\" | \"groq\" | \"github\" => {\n let base = match provider {\n \"openai\" => \"https://api.openai.com/v1\",\n \"xai\" => \"https://api.x.ai/v1\",\n \"fireworks\" => \"https://api.fireworks.ai/inference/v1\",\n \"deepseek\" => \"https://api.deepseek.com/v1\",\n \"mistral\" => \"https://api.mistral.ai/v1\",\n \"groq\" => \"https://api.groq.com/openai/v1\",\n \"github\" => \"https://models.inference.ai.azure.com\",\n _ => unreachable!(),\n };\n Ok((\n format!(\"{}/chat/completions\", base),\n json!({ \"model\": model_name, \"messages\": [{\"role\":\"user\",\"content\": prompt}], \"max_tokens\": 2048 })\n ))\n}\n```\n\n- [ ] **Step 4: Add cloudflare to `build_api_request`**\n\nAfter the `\"google\" => ...` arm (line ~206), add:\n\n```rust\n\"cloudflare\" => {\n let account_id = std::env::var(\"CF_ACCOUNT_ID\").unwrap_or_default();\n Ok((\n format!(\n \"https://api.cloudflare.com/client/v4/accounts/{}/ai/run/{}\",\n account_id, model_name\n ),\n json!({ \"messages\": [{\"role\":\"user\",\"content\": prompt}], \"max_tokens\": 2048 })\n ))\n},\n```\n\n- [ ] **Step 5: Add cloudflare to `extract_code`**\n\nIn `extract_code`, add a `\"cloudflare\"` arm before the default `_` arm:\n\n```rust\nfn extract_code(resp: &Value, provider: &str) -> Result<String> {\n let text = match provider {\n \"anthropic\" => resp[\"content\"][0][\"text\"].as_str().unwrap_or(\"\"),\n \"google\" => resp[\"candidates\"][0][\"content\"][\"parts\"][0][\"text\"].as_str().unwrap_or(\"\"),\n \"qwen\" => resp[\"output\"][\"text\"].as_str().unwrap_or(\"\"),\n \"hunyuan\" => resp[\"Choices\"][0][\"Message\"][\"Content\"].as_str().unwrap_or(\"\"),\n \"cloudflare\" => resp[\"result\"][\"response\"].as_str().unwrap_or(\"\"),\n _ => resp[\"choices\"][0][\"message\"][\"content\"].as_str().unwrap_or(\"\"),\n };\n // ... rest of function unchanged\n```\n\n- [ ] **Step 6: Run tests to verify they pass**\n\n```bash\ncargo test --manifest-path backend/Cargo.toml 2>&1 | tail -20\n```\n\nExpected: all new tests pass. Note: `test_build_api_request_cloudflare` sets `CF_ACCOUNT_ID` env var — if tests run in parallel this is fine since each test sets it before asserting.\n\n- [ ] **Step 7: Commit**\n\n```bash\ngit add backend/src/runner.rs\ngit commit -m \"feat: add groq, github, cloudflare provider support\"\n```\n\n---\n\n### Task 2: Add config fields for new providers\n\n**Files:**\n- Modify: `backend/src/config.rs`\n\n**Context:** `config.rs` holds all API keys loaded from env vars on startup. The `runner.rs` reads API keys at race time via `std::env::var(&model.api_key_env)`, so these config fields aren't strictly required for the runner — but they're needed for the `sync_models` auto-discovery function in `sync.rs` (which checks `if !config.groq_api_key.is_empty()`). Add them now so `sync.rs` can be extended in the future. `CF_ACCOUNT_ID` is used directly in `build_api_request` via `std::env::var`, so it doesn't need a config field — but add it for completeness and testability.\n\n- [ ] **Step 1: Write failing test**\n\nAdd to `backend/src/config.rs`:\n\n```rust\n#[cfg(test)]\nmod tests {\n use super::*;\n\n #[test]\n fn test_config_new_fields_default_empty() {\n // Ensure new fields don't panic when env vars are absent\n std::env::remove_var(\"GROQ_API_KEY\");\n std::env::remove_var(\"GITHUB_TOKEN\");\n std::env::remove_var(\"CF_API_TOKEN\");\n std::env::remove_var(\"CF_ACCOUNT_ID\");\n let cfg = Config::from_env().unwrap();\n assert_eq!(cfg.groq_api_key, \"\");\n assert_eq!(cfg.github_token, \"\");\n assert_eq!(cfg.cf_api_token, \"\");\n assert_eq!(cfg.cf_account_id, \"\");\n }\n}\n```\n\n- [ ] **Step 2: Run test to verify it fails**\n\n```bash\ncargo test --manifest-path backend/Cargo.toml config::tests 2>&1\n```\n\nExpected: compile error — fields don't exist yet.\n\n- [ ] **Step 3: Add fields to Config struct**\n\nIn `backend/src/config.rs`, add to the `Config` struct after `mistral_api_key`:\n\n```rust\npub groq_api_key: String,\npub github_token: String,\npub cf_api_token: String,\npub cf_account_id: String,\n```\n\n- [ ] **Step 4: Add fields to `from_env`**\n\nIn `from_env()`, add after `mistral_api_key: std::env::var(\"MISTRAL_API_KEY\").unwrap_or_default(),`:\n\n```rust\ngroq_api_key: std::env::var(\"GROQ_API_KEY\").unwrap_or_default(),\ngithub_token: std::env::var(\"GITHUB_TOKEN\").unwrap_or_default(),\ncf_api_token: std::env::var(\"CF_API_TOKEN\").unwrap_or_default(),\ncf_account_id: std::env::var(\"CF_ACCOUNT_ID\").unwrap_or_default(),\n```\n\n- [ ] **Step 5: Run test to verify it passes**\n\n```bash\ncargo test --manifest-path backend/Cargo.toml config::tests 2>&1\n```\n\nExpected: PASS.\n\n- [ ] **Step 6: Commit**\n\n```bash\ngit add backend/src/config.rs\ngit commit -m \"feat: add groq, github, cloudflare fields to Config\"\n```\n\n---\n\n### Task 3: Fix seed and add new model rows\n\n**Files:**\n- Modify: `backend/src/sync.rs`\n\n**Context:** `seed_initial_models` currently short-circuits with `if count > 0 { return Ok(()); }` and uses a plain INSERT with no conflict handling. This means new model rows never get added to existing deployments. Fix: remove the count guard and add `ON CONFLICT (name) DO NOTHING` to the INSERT so it's safe to run on every startup. Then add the 5 new model rows (groq ×2, github ×2, cloudflare ×1). Note: gemini-2.0-flash, deepseek-chat, deepseek-reasoner, and qwen models are already in the seed list.\n\n- [ ] **Step 1: Remove the count guard**\n\nIn `backend/src/sync.rs`, find `seed_initial_models`. Remove these lines:\n\n```rust\nlet count: i64 = sqlx::query_scalar!(\"SELECT COUNT(*) FROM models\")\n .fetch_one(pool).await?.unwrap_or(0);\n\nif count > 0 { return Ok(()); }\n```\n\n- [ ] **Step 2: Add `ON CONFLICT (name) DO NOTHING` to the seed INSERT**\n\nFind the `sqlx::query!` call inside the `for` loop in `seed_initial_models`. Change:\n\n```rust\nsqlx::query!(\n \"INSERT INTO models (id, provider, name, display_name, api_key_env, is_active, is_new, is_human, human_times, added_at) VALUES ($1, $2, $3, $4, $5, true, false, $6, $7, $8)\",\n id, provider, name, display, key_env, is_human, human_times, now\n).execute(pool).await?;\n```\n\nTo:\n\n```rust\nsqlx::query!(\n \"INSERT INTO models (id, provider, name, display_name, api_key_env, is_active, is_new, is_human, human_times, added_at) VALUES ($1, $2, $3, $4, $5, true, false, $6, $7, $8) ON CONFLICT (name) DO NOTHING\",\n id, provider, name, display, key_env, is_human, human_times, now\n).execute(pool).await?;\n```\n\n- [ ] **Step 3: Add new model rows to the seed list**\n\nFind the `models: &[(&str, &str, &str, &str, bool)]` slice in `seed_initial_models`. Add these 5 rows after the existing entries and before the human rows:\n\n```rust\n(\"groq\", \"llama-3.3-70b-versatile\", \"Llama 3.3 70B\", \"GROQ_API_KEY\", false),\n(\"groq\", \"mixtral-8x7b-32768\", \"Mixtral 8x7B\", \"GROQ_API_KEY\", false),\n(\"github\", \"gpt-4o-mini\", \"GPT-4o mini\", \"GITHUB_TOKEN\", false),\n(\"github\", \"Meta-Llama-3-70B-Instruct\", \"Llama 3 70B\", \"GITHUB_TOKEN\", false),\n(\"cloudflare\", \"@cf/meta/llama-3.1-8b-instruct\", \"Llama 3.1 8B\", \"CF_API_TOKEN\", false),\n```\n\n- [ ] **Step 4: Verify compilation**\n\n```bash\ncargo check --manifest-path backend/Cargo.toml 2>&1\n```\n\nExpected: no errors. (sqlx compile-time query checks run against `DATABASE_URL` — if not set locally, use `SQLX_OFFLINE=true cargo check ...`)\n\n- [ ] **Step 5: Commit**\n\n```bash\ngit add backend/src/sync.rs\ngit commit -m \"feat: seed groq, github, cloudflare models; always upsert on startup\"\n```\n\n---\n\n### Task 4: On-demand benchmark trigger in problems.rs\n\n**Files:**\n- Modify: `backend/src/routes/problems.rs`\n\n**Context:** The `results` handler currently fetches and returns race results for a problem. We extend it to: (1) check which active non-human models have no result for this problem, (2) if any are missing, spawn a `tokio::spawn` background task that calls `runner.race()` and saves results. The HTTP response returns immediately with existing results — the frontend's polling loop surfaces new results as they complete. `AppState` already has `runner: Arc<Runner>` and `pool: PgPool`.\n\n- [ ] **Step 1: Write a test for the missing-model detection logic**\n\nThe detection logic (finding model IDs with no results) is pure set math. Write a unit test for it:\n\n```rust\n#[cfg(test)]\nmod tests {\n use std::collections::HashSet;\n\n #[test]\n fn test_missing_model_ids() {\n let benchmarked: HashSet<String> = [\"model-a\", \"model-b\"]\n .iter().map(|s| s.to_string()).collect();\n let all_active = vec![\"model-a\", \"model-b\", \"model-c\", \"model-d\"];\n let missing: Vec<&str> = all_active.into_iter()\n .filter(|id| !benchmarked.contains(*id))\n .collect();\n assert_eq!(missing, vec![\"model-c\", \"model-d\"]);\n }\n\n #[test]\n fn test_no_missing_models_when_all_benchmarked() {\n let benchmarked: HashSet<String> = [\"model-a\"].iter().map(|s| s.to_string()).collect();\n let all_active = vec![\"model-a\"];\n let missing: Vec<&str> = all_active.into_iter()\n .filter(|id| !benchmarked.contains(*id))\n .collect();\n assert!(missing.is_empty());\n }\n}\n```\n\nAdd this at the bottom of `backend/src/routes/problems.rs`.\n\n- [ ] **Step 2: Run tests to verify they pass (pure logic, no DB needed)**\n\n```bash\ncargo test --manifest-path backend/Cargo.toml routes::problems 2>&1\n```\n\nExpected: 2 tests pass.\n\n- [ ] **Step 3: Add imports to problems.rs**\n\nAt the top of `backend/src/routes/problems.rs`, add to the existing `use` block:\n\n```rust\nuse std::collections::HashSet;\nuse tokio::sync::broadcast;\nuse uuid::Uuid;\nuse crate::models::Model;\n```\n\n- [ ] **Step 4: Replace the `results` handler with the on-demand trigger version**\n\nReplace the entire `results` function in `backend/src/routes/problems.rs`:\n\n```rust\npub async fn results(\n State(state): State<AppState>,\n Path(id): Path<String>,\n) -> Json<Vec<RaceResultWithModel>> {\n let rows = sqlx::query!(\n r#\"SELECT r.id, r.problem_id, r.model_id, r.solved, r.time_ms, r.attempts, r.run_at,\n m.display_name, m.provider, m.name as model_name, m.is_human\n FROM results r JOIN models m ON r.model_id = m.id\n WHERE r.problem_id = $1\n ORDER BY r.time_ms ASC NULLS LAST\"#,\n id\n ).fetch_all(&state.pool).await.unwrap_or_default();\n\n let results: Vec<RaceResultWithModel> = rows.into_iter().map(|r| RaceResultWithModel {\n model_id: r.model_id,\n model_name: r.model_name,\n display_name: r.display_name,\n provider: r.provider,\n is_human: r.is_human,\n solved: r.solved,\n time_ms: r.time_ms,\n attempts: r.attempts,\n run_at: r.run_at,\n }).collect();\n\n // On-demand trigger: find active models with no result for this problem\n let benchmarked_ids: HashSet<String> = results.iter()\n .filter(|r| !r.is_human)\n .map(|r| r.model_id.clone())\n .collect();\n\n let all_active = sqlx::query_as!(\n Model,\n \"SELECT * FROM models WHERE is_active = true AND is_human = false\"\n ).fetch_all(&state.pool).await.unwrap_or_default();\n\n let missing: Vec<Model> = all_active.into_iter()\n .filter(|m| !benchmarked_ids.contains(&m.id))\n .collect();\n\n if !missing.is_empty() {\n let problem_result = sqlx::query_as!(\n crate::models::Problem,\n \"SELECT * FROM problems WHERE id = $1\",\n id\n ).fetch_optional(&state.pool).await.unwrap_or(None);\n\n if let Some(problem) = problem_result {\n let runner = state.runner.clone();\n let pool = state.pool.clone();\n let race_id = Uuid::new_v4().to_string();\n let (tx, _) = broadcast::channel(64);\n tokio::spawn(async move {\n tracing::info!(\n \"on-demand benchmark: {} missing models for problem {}\",\n missing.len(), problem.id\n );\n let results = runner.race(&race_id, &problem, missing, tx).await;\n for result in &results {\n sqlx::query!(\n r#\"INSERT INTO results (id, problem_id, model_id, solved, time_ms, attempts, run_at)\n VALUES ($1, $2, $3, $4, $5, $6, $7)\n ON CONFLICT (id) DO NOTHING\"#,\n result.id, result.problem_id, result.model_id,\n result.solved, result.time_ms, result.attempts, result.run_at\n ).execute(&pool).await.ok();\n }\n });\n }\n }\n\n Json(results)\n}\n```\n\n- [ ] **Step 5: Verify compilation**\n\n```bash\ncargo check --manifest-path backend/Cargo.toml 2>&1\n```\n\nExpected: no errors.\n\n- [ ] **Step 6: Commit**\n\n```bash\ngit add backend/src/routes/problems.rs\ngit commit -m \"feat: on-demand benchmark trigger when models are missing results\"\n```\n\n---\n\n### Task 5: Document new env vars\n\n**Files:**\n- Modify: `backend/.env.example`\n\n- [ ] **Step 1: Add new env vars to .env.example**\n\nOpen `backend/.env.example` and add after the existing key entries:\n\n```\n# Free-tier providers\nGROQ_API_KEY=\nGITHUB_TOKEN=\nCF_API_TOKEN=\nCF_ACCOUNT_ID=\n```\n\n- [ ] **Step 2: Verify no other changes needed**\n\n```bash\ncargo check --manifest-path backend/Cargo.toml 2>&1\n```\n\nExpected: clean.\n\n- [ ] **Step 3: Commit**\n\n```bash\ngit add backend/.env.example\ngit commit -m \"docs: add free provider env vars to .env.example\"\n```\n\n---\n\n## Post-implementation\n\nAfter all tasks are complete, set env vars in Railway for any providers you have keys for:\n\n```\nGROQ_API_KEY=<from console.groq.com>\nGITHUB_TOKEN=<GitHub personal access token with Models permission>\nCF_API_TOKEN=<from Cloudflare dashboard>\nCF_ACCOUNT_ID=<from Cloudflare dashboard>\n```\n\nOn next deploy, `seed_initial_models` will insert the 5 new model rows. On first user visit to any problem, the on-demand trigger fires and benchmarks all missing models in the background.\n"}}}} |