1 line
22 KiB
JSON
1 line
22 KiB
JSON
{"repo": "chad/whichlang", "files": {"plan.md": {"c70e2457d3322c98c509c3b5c6e95f9220ba38a7": {"t": 1785747213.3876817, "n": 8385, "h": "e80cc12e2dfea131", "c": "# whichlang — plan\n\n## Goal\n\nA harness that asks LLMs to write code for common, language-agnostic tasks and records which programming language each model defaults to. Output is a table (model × task → language distribution) that's useful to developers as a signal of what an LLM will reach for unprompted.\n\n## Design decisions (from user)\n\n- **Sampling**: N runs per (model, task), report full distribution (e.g. `python 7/10, go 3/10`).\n- **Language detection**: judge LLM classifies each response (handles weird formatting, multi-language responses, \"I'd recommend X\" prose).\n- **Task categories**: scripting/glue, backend service/API, CLI tool, web app.\n- **Output**: markdown table for humans + JSONL of raw runs so others can re-aggregate.\n\n## Architecture\n\n```\nwhichlang/\n├── tasks.yaml # ~3-5 prompts per category, language-neutral\n├── models.yaml # list of (provider, model_id, display_name)\n├── whichlang/\n│ ├── providers.py # unified .complete(model, prompt) → str across Anthropic, OpenAI, Google, OpenAI-compatible\n│ ├── classify.py # judge LLM: response → language name (lowercase) or \"none\"\n│ ├── run.py # main loop; resumable; appends to results/runs.jsonl\n│ └── report.py # JSONL → REPORT.md (markdown table) + summary stats\n├── results/runs.jsonl # one line per (model, task, sample) with raw response + classified language\n└── REPORT.md # generated\n```\n\n### Provider abstraction\n\nOne function: `complete(model_spec, prompt) -> str`. `model_spec` carries provider + model_id + optional base_url. This makes adding open models (DeepSeek, Llama, Qwen via Together/Fireworks/OpenRouter) a config-file change, not a code change — any OpenAI-compatible endpoint plugs in.\n\n### Resumability\n\nRuns.jsonl is append-only. Each line carries `(model_id, task_id, sample_idx)`. On startup, `run.py` reads existing lines and skips combinations already done. Safe to ctrl-C and resume.\n\n### Prompt design\n\nPrompts must NOT mention a language or ask the model to choose. They should read like a normal developer request: \"I need a tool that takes a CSV and emits JSON. Write it.\" If we hint or ask \"what language would you use,\" we measure something different.\n\nSystem prompt: minimal — just nudge toward producing runnable code in one language, no clarifying questions. Goal is to capture the model's default, not its question-asking behavior.\n\n### Judge\n\nCheap small model (e.g. claude-haiku or gpt-5-mini). Input: the raw response. Output: a single lowercase language token. Bias check: judge sees only the code, not which model produced it.\n\n## Initial model set\n\nAdjustable in models.yaml. Starting with:\n- Anthropic: claude-opus-4-7, claude-sonnet-4-6, claude-haiku-4-5\n- OpenAI: gpt-5, gpt-5-mini (or current equivalents)\n- Google: gemini-2.5-pro, gemini-2.5-flash\n\n## Steps\n\n1. [x] Confirm design with user\n2. [x] Write providers.py\n3. [x] Write tasks.yaml (16 tasks across 4 categories)\n4. [x] Write models.yaml (3 Anthropic + 2 OpenAI + 2 Google = 7 models)\n5. [x] Write run.py (resumable loop, JSONL output)\n6. [x] Write classify.py (judge LLM = claude-haiku-4-5)\n7. [x] Write report.py (markdown table generator)\n8. [x] Smoke test: 1 sample on Anthropic + OpenAI + Google paths — all three work end-to-end\n9. [x] README\n10. [x] Initial git commit\n11. [ ] Ask user before running the full benchmark (API spend ~7×16×5 = 560 generation calls + 560 judge calls)\n\n## Notes from smoke test\n\n- Resume logic initially counted errored runs as \"done\", so a transient Google 503 would\n never get retried. Fixed: errors stay in the JSONL for the record but `_load_done`\n ignores them so they get re-attempted on the next run.\n- All three providers wire up cleanly. The classifier returned \"python\" on the same task\n for all three smoke-tested models (csv_to_json), which is the expected boring baseline.\n\n## Open questions to revisit\n\n- How many samples per (model, task)? Starting plan: 5. Trades cost vs. distribution confidence.\n- Do we also capture *reasoning* (does the model justify the choice)? Could be a follow-up column.\n- Should the prompt forbid clarifying questions or let \"I'd need to know X first\" responses count as a no-op? Currently treating no-code responses as `none` in the distribution.\n\n---\n\n# Next: add open models\n\nThe harness already supports any OpenAI-compatible endpoint, so adding open models is a\nYAML edit, not a code change. The interesting decision is which ones, and via what host.\n\n## Recommended open models to add (in priority order)\n\n1. **DeepSeek V3.1** (or V3.2 if released) — frontier-competitive, MIT/MIT-ish license,\n wide deployment. Has both a reasoning (\"R1\"-style) and non-reasoning variant.\n2. **Qwen 3 Coder** (32B or 480B if hosted) — Alibaba's coding-tuned line is the open\n model most likely to behave *differently* from Western frontier models on language\n choice. If anything is going to default to Java or C++, it's a Chinese coder model.\n3. **Llama 3.3 70B Instruct** (or Llama 4 Maverick if released and stable) — Meta's\n flagship instruct line, widely cited as the open baseline.\n\nStretch picks (round it out to 5 if budget allows):\n- **Mistral Large 2** — European frontier-tier open weights.\n- **Gemma 3 27B** — Google's open line, comparator for Gemini.\n- **GLM 4** — increasingly cited.\n\n## How to serve them\n\n### Recommended: OpenRouter (hosted, single API key, all three above)\n\n- One key (`OPENROUTER_API_KEY`), one base_url (`https://openrouter.ai/api/v1`), routes\n to DeepSeek / Qwen / Llama and dozens more.\n- Full-precision weights (no quantization artifacts skewing defaults).\n- Pay-as-you-go, no monthly minimum. Whole 3-model × 16-task × 5-sample run would be\n under $1 for these models (they're cheap on OpenRouter).\n- Slots straight into our `openai_compatible` provider — already supported, no new code.\n\n### Alternative: Together AI / Fireworks / DeepInfra\n\n- Same OpenAI-compatible shape, separate API keys per host.\n- Sometimes lower latency or better availability than OpenRouter for a specific model.\n- Worth using as a fallback when OpenRouter is rate-limiting.\n\n### Local: Ollama (for reproducibility, smaller models)\n\n- User has Ollama installed (0.5.3 — current is 0.6.x+, **update first**:\n `curl https://ollama.com/install.sh | sh` or download from ollama.com).\n- Daemon needs to be running (`ollama serve` or launch the Mac app).\n- Exposes an OpenAI-compatible endpoint at `http://localhost:11434/v1` — drops into our\n `openai_compatible` provider, set `api_key_env` to a dummy (Ollama doesn't check it).\n- Quantization caveat: Ollama defaults to Q4_K_M (4-bit). For a \"what does this model\n default to?\" benchmark this may not matter much, but it's worth noting in REPORT.md.\n- Hardware limit: 70B models need ~40GB+ RAM. 7B–32B models run well on M-series Macs.\n For Chad's machine, realistic Ollama set: qwen3-coder (7B/32B), llama3.3 (depending\n on RAM), deepseek-r1:32b (the distilled variant).\n\n## Plan\n\n1. [ ] Update Ollama to 0.6.x+ and start the daemon\n2. [ ] Decide: hosted (OpenRouter, full-precision) vs local (Ollama, quantized) vs both\n3. [ ] Add 3 open models to `models.yaml` based on the chosen host\n4. [ ] Run benchmark on just those models (`--models <ids>`); ~$0.50–1.00 on OpenRouter\n5. [ ] Regenerate REPORT.md; commit JSONL + updated report\n6. [ ] Document quantization caveat in README if Ollama path used\n7. [ ] Retry Gemini once free-tier quota resets (or move to billed key) so the table is\n complete\n\n## Why these picks specifically?\n\nThe benchmark is most interesting when the model set *spans the space of defaults*.\nThree Anthropic + two OpenAI models all defaulted to Python — not surprising, similar\ntraining data. The open model picks above are chosen to **stress-test that assumption**:\n\n- **DeepSeek** is Chinese training data, MoE architecture, distinct lineage.\n- **Qwen Coder** is explicitly coding-tuned and Chinese-origin — most likely to surface\n a different default if one exists in the open-model space.\n- **Llama** is the most-deployed open baseline; differences from frontier models tell us\n whether \"the default is Python\" is a frontier phenomenon or universal.\n\nIf all three also default to Python, that's itself the story: it's not training-data bias\nor RLHF preference, it's that Python *is* the right answer for these prompts.\n"}, "bd2864f6b8fc6c43ea030ebb1374315e95071042": {"t": 1785771409.7356668, "n": 8385, "h": "e80cc12e2dfea131", "c": "# whichlang — plan\n\n## Goal\n\nA harness that asks LLMs to write code for common, language-agnostic tasks and records which programming language each model defaults to. Output is a table (model × task → language distribution) that's useful to developers as a signal of what an LLM will reach for unprompted.\n\n## Design decisions (from user)\n\n- **Sampling**: N runs per (model, task), report full distribution (e.g. `python 7/10, go 3/10`).\n- **Language detection**: judge LLM classifies each response (handles weird formatting, multi-language responses, \"I'd recommend X\" prose).\n- **Task categories**: scripting/glue, backend service/API, CLI tool, web app.\n- **Output**: markdown table for humans + JSONL of raw runs so others can re-aggregate.\n\n## Architecture\n\n```\nwhichlang/\n├── tasks.yaml # ~3-5 prompts per category, language-neutral\n├── models.yaml # list of (provider, model_id, display_name)\n├── whichlang/\n│ ├── providers.py # unified .complete(model, prompt) → str across Anthropic, OpenAI, Google, OpenAI-compatible\n│ ├── classify.py # judge LLM: response → language name (lowercase) or \"none\"\n│ ├── run.py # main loop; resumable; appends to results/runs.jsonl\n│ └── report.py # JSONL → REPORT.md (markdown table) + summary stats\n├── results/runs.jsonl # one line per (model, task, sample) with raw response + classified language\n└── REPORT.md # generated\n```\n\n### Provider abstraction\n\nOne function: `complete(model_spec, prompt) -> str`. `model_spec` carries provider + model_id + optional base_url. This makes adding open models (DeepSeek, Llama, Qwen via Together/Fireworks/OpenRouter) a config-file change, not a code change — any OpenAI-compatible endpoint plugs in.\n\n### Resumability\n\nRuns.jsonl is append-only. Each line carries `(model_id, task_id, sample_idx)`. On startup, `run.py` reads existing lines and skips combinations already done. Safe to ctrl-C and resume.\n\n### Prompt design\n\nPrompts must NOT mention a language or ask the model to choose. They should read like a normal developer request: \"I need a tool that takes a CSV and emits JSON. Write it.\" If we hint or ask \"what language would you use,\" we measure something different.\n\nSystem prompt: minimal — just nudge toward producing runnable code in one language, no clarifying questions. Goal is to capture the model's default, not its question-asking behavior.\n\n### Judge\n\nCheap small model (e.g. claude-haiku or gpt-5-mini). Input: the raw response. Output: a single lowercase language token. Bias check: judge sees only the code, not which model produced it.\n\n## Initial model set\n\nAdjustable in models.yaml. Starting with:\n- Anthropic: claude-opus-4-7, claude-sonnet-4-6, claude-haiku-4-5\n- OpenAI: gpt-5, gpt-5-mini (or current equivalents)\n- Google: gemini-2.5-pro, gemini-2.5-flash\n\n## Steps\n\n1. [x] Confirm design with user\n2. [x] Write providers.py\n3. [x] Write tasks.yaml (16 tasks across 4 categories)\n4. [x] Write models.yaml (3 Anthropic + 2 OpenAI + 2 Google = 7 models)\n5. [x] Write run.py (resumable loop, JSONL output)\n6. [x] Write classify.py (judge LLM = claude-haiku-4-5)\n7. [x] Write report.py (markdown table generator)\n8. [x] Smoke test: 1 sample on Anthropic + OpenAI + Google paths — all three work end-to-end\n9. [x] README\n10. [x] Initial git commit\n11. [ ] Ask user before running the full benchmark (API spend ~7×16×5 = 560 generation calls + 560 judge calls)\n\n## Notes from smoke test\n\n- Resume logic initially counted errored runs as \"done\", so a transient Google 503 would\n never get retried. Fixed: errors stay in the JSONL for the record but `_load_done`\n ignores them so they get re-attempted on the next run.\n- All three providers wire up cleanly. The classifier returned \"python\" on the same task\n for all three smoke-tested models (csv_to_json), which is the expected boring baseline.\n\n## Open questions to revisit\n\n- How many samples per (model, task)? Starting plan: 5. Trades cost vs. distribution confidence.\n- Do we also capture *reasoning* (does the model justify the choice)? Could be a follow-up column.\n- Should the prompt forbid clarifying questions or let \"I'd need to know X first\" responses count as a no-op? Currently treating no-code responses as `none` in the distribution.\n\n---\n\n# Next: add open models\n\nThe harness already supports any OpenAI-compatible endpoint, so adding open models is a\nYAML edit, not a code change. The interesting decision is which ones, and via what host.\n\n## Recommended open models to add (in priority order)\n\n1. **DeepSeek V3.1** (or V3.2 if released) — frontier-competitive, MIT/MIT-ish license,\n wide deployment. Has both a reasoning (\"R1\"-style) and non-reasoning variant.\n2. **Qwen 3 Coder** (32B or 480B if hosted) — Alibaba's coding-tuned line is the open\n model most likely to behave *differently* from Western frontier models on language\n choice. If anything is going to default to Java or C++, it's a Chinese coder model.\n3. **Llama 3.3 70B Instruct** (or Llama 4 Maverick if released and stable) — Meta's\n flagship instruct line, widely cited as the open baseline.\n\nStretch picks (round it out to 5 if budget allows):\n- **Mistral Large 2** — European frontier-tier open weights.\n- **Gemma 3 27B** — Google's open line, comparator for Gemini.\n- **GLM 4** — increasingly cited.\n\n## How to serve them\n\n### Recommended: OpenRouter (hosted, single API key, all three above)\n\n- One key (`OPENROUTER_API_KEY`), one base_url (`https://openrouter.ai/api/v1`), routes\n to DeepSeek / Qwen / Llama and dozens more.\n- Full-precision weights (no quantization artifacts skewing defaults).\n- Pay-as-you-go, no monthly minimum. Whole 3-model × 16-task × 5-sample run would be\n under $1 for these models (they're cheap on OpenRouter).\n- Slots straight into our `openai_compatible` provider — already supported, no new code.\n\n### Alternative: Together AI / Fireworks / DeepInfra\n\n- Same OpenAI-compatible shape, separate API keys per host.\n- Sometimes lower latency or better availability than OpenRouter for a specific model.\n- Worth using as a fallback when OpenRouter is rate-limiting.\n\n### Local: Ollama (for reproducibility, smaller models)\n\n- User has Ollama installed (0.5.3 — current is 0.6.x+, **update first**:\n `curl https://ollama.com/install.sh | sh` or download from ollama.com).\n- Daemon needs to be running (`ollama serve` or launch the Mac app).\n- Exposes an OpenAI-compatible endpoint at `http://localhost:11434/v1` — drops into our\n `openai_compatible` provider, set `api_key_env` to a dummy (Ollama doesn't check it).\n- Quantization caveat: Ollama defaults to Q4_K_M (4-bit). For a \"what does this model\n default to?\" benchmark this may not matter much, but it's worth noting in REPORT.md.\n- Hardware limit: 70B models need ~40GB+ RAM. 7B–32B models run well on M-series Macs.\n For Chad's machine, realistic Ollama set: qwen3-coder (7B/32B), llama3.3 (depending\n on RAM), deepseek-r1:32b (the distilled variant).\n\n## Plan\n\n1. [ ] Update Ollama to 0.6.x+ and start the daemon\n2. [ ] Decide: hosted (OpenRouter, full-precision) vs local (Ollama, quantized) vs both\n3. [ ] Add 3 open models to `models.yaml` based on the chosen host\n4. [ ] Run benchmark on just those models (`--models <ids>`); ~$0.50–1.00 on OpenRouter\n5. [ ] Regenerate REPORT.md; commit JSONL + updated report\n6. [ ] Document quantization caveat in README if Ollama path used\n7. [ ] Retry Gemini once free-tier quota resets (or move to billed key) so the table is\n complete\n\n## Why these picks specifically?\n\nThe benchmark is most interesting when the model set *spans the space of defaults*.\nThree Anthropic + two OpenAI models all defaulted to Python — not surprising, similar\ntraining data. The open model picks above are chosen to **stress-test that assumption**:\n\n- **DeepSeek** is Chinese training data, MoE architecture, distinct lineage.\n- **Qwen Coder** is explicitly coding-tuned and Chinese-origin — most likely to surface\n a different default if one exists in the open-model space.\n- **Llama** is the most-deployed open baseline; differences from frontier models tell us\n whether \"the default is Python\" is a frontier phenomenon or universal.\n\nIf all three also default to Python, that's itself the story: it's not training-data bias\nor RLHF preference, it's that Python *is* the right answer for these prompts.\n"}, "0958248bc75933acc25f34b6cb1dbe89ef96c41f": {"t": 1785771411.09959, "n": 4305, "h": "c90280b4f2a00709", "c": "# whichlang — plan\n\n## Goal\n\nA harness that asks LLMs to write code for common, language-agnostic tasks and records which programming language each model defaults to. Output is a table (model × task → language distribution) that's useful to developers as a signal of what an LLM will reach for unprompted.\n\n## Design decisions (from user)\n\n- **Sampling**: N runs per (model, task), report full distribution (e.g. `python 7/10, go 3/10`).\n- **Language detection**: judge LLM classifies each response (handles weird formatting, multi-language responses, \"I'd recommend X\" prose).\n- **Task categories**: scripting/glue, backend service/API, CLI tool, web app.\n- **Output**: markdown table for humans + JSONL of raw runs so others can re-aggregate.\n\n## Architecture\n\n```\nwhichlang/\n├── tasks.yaml # ~3-5 prompts per category, language-neutral\n├── models.yaml # list of (provider, model_id, display_name)\n├── whichlang/\n│ ├── providers.py # unified .complete(model, prompt) → str across Anthropic, OpenAI, Google, OpenAI-compatible\n│ ├── classify.py # judge LLM: response → language name (lowercase) or \"none\"\n│ ├── run.py # main loop; resumable; appends to results/runs.jsonl\n│ └── report.py # JSONL → REPORT.md (markdown table) + summary stats\n├── results/runs.jsonl # one line per (model, task, sample) with raw response + classified language\n└── REPORT.md # generated\n```\n\n### Provider abstraction\n\nOne function: `complete(model_spec, prompt) -> str`. `model_spec` carries provider + model_id + optional base_url. This makes adding open models (DeepSeek, Llama, Qwen via Together/Fireworks/OpenRouter) a config-file change, not a code change — any OpenAI-compatible endpoint plugs in.\n\n### Resumability\n\nRuns.jsonl is append-only. Each line carries `(model_id, task_id, sample_idx)`. On startup, `run.py` reads existing lines and skips combinations already done. Safe to ctrl-C and resume.\n\n### Prompt design\n\nPrompts must NOT mention a language or ask the model to choose. They should read like a normal developer request: \"I need a tool that takes a CSV and emits JSON. Write it.\" If we hint or ask \"what language would you use,\" we measure something different.\n\nSystem prompt: minimal — just nudge toward producing runnable code in one language, no clarifying questions. Goal is to capture the model's default, not its question-asking behavior.\n\n### Judge\n\nCheap small model (e.g. claude-haiku or gpt-5-mini). Input: the raw response. Output: a single lowercase language token. Bias check: judge sees only the code, not which model produced it.\n\n## Initial model set\n\nAdjustable in models.yaml. Starting with:\n- Anthropic: claude-opus-4-7, claude-sonnet-4-6, claude-haiku-4-5\n- OpenAI: gpt-5, gpt-5-mini (or current equivalents)\n- Google: gemini-2.5-pro, gemini-2.5-flash\n\n## Steps\n\n1. [x] Confirm design with user\n2. [x] Write providers.py\n3. [x] Write tasks.yaml (16 tasks across 4 categories)\n4. [x] Write models.yaml (3 Anthropic + 2 OpenAI + 2 Google = 7 models)\n5. [x] Write run.py (resumable loop, JSONL output)\n6. [x] Write classify.py (judge LLM = claude-haiku-4-5)\n7. [x] Write report.py (markdown table generator)\n8. [x] Smoke test: 1 sample on Anthropic + OpenAI + Google paths — all three work end-to-end\n9. [x] README\n10. [x] Initial git commit\n11. [ ] Ask user before running the full benchmark (API spend ~7×16×5 = 560 generation calls + 560 judge calls)\n\n## Notes from smoke test\n\n- Resume logic initially counted errored runs as \"done\", so a transient Google 503 would\n never get retried. Fixed: errors stay in the JSONL for the record but `_load_done`\n ignores them so they get re-attempted on the next run.\n- All three providers wire up cleanly. The classifier returned \"python\" on the same task\n for all three smoke-tested models (csv_to_json), which is the expected boring baseline.\n\n## Open questions to revisit\n\n- How many samples per (model, task)? Starting plan: 5. Trades cost vs. distribution confidence.\n- Do we also capture *reasoning* (does the model justify the choice)? Could be a follow-up column.\n- Should the prompt forbid clarifying questions or let \"I'd need to know X first\" responses count as a no-op? Currently treating no-code responses as `none` in the distribution.\n"}}}} |