1 line
6.0 KiB
JSON
1 line
6.0 KiB
JSON
{"repo": "0xzr/freellmpool", "files": {"docs/MODEL_ACTIVITY_AUDIT_2026-07-29.md": {"172b1bb6759a1c08cb9d9f7d4e247ca05b34126c": {"t": 1785596007.352422, "n": 5853, "h": "da1551a3e209c39e", "c": "# Exhaustive model activity audit — 2026-07-29\n\nThis audit sent only fixed synthetic canaries through freellmpool's real client\npaths. Chat used `Reply with the single word: pong`, embeddings used\n`fixed catalog audit canary`, and transcription used a generated 0.25-second\n440 Hz WAV. No repository content, user prompt, credential value, or\naccount-specific diagnostic identifier is retained here.\n\nRaw reports remain under `/tmp` because upstream diagnostics can contain\naccount-specific identifiers. This document contains only sanitized aggregates\nand catalog decisions.\n\n## Coverage and method\n\n- **383 of 407 chat routes were live-probed**, including enabled and disabled\n routes on all 18 configured providers.\n- The 24 routes without a usable maintenance configuration are provider-specific:\n Aion Labs (5; no `AION_API_KEY`), ModelScope (7; no\n `MODELSCOPE_API_KEY`), Morph (3; no `MORPH_API_KEY`), Vercel AI Gateway\n (5; its billing-safety opt-in was deliberately absent), SiliconFlow (1; no\n `SILICONFLOW_API_KEY`), and Gemini (3 cataloged, including two disabled\n lifecycle rows; no `GEMINI_API_KEY`).\n- Transient HTTP statuses and transport failures received up to three attempts.\n Every enabled route classified as definitively unavailable was then called\n three more times independently. All 25 repeated the same definitive\n 400/404/410 classification before being disabled.\n- A 401, 402, 429, timeout, provider-wide quota error, or isolated 5xx did not\n disable any route.\n- All **21 enabled embedding routes** and **5 enabled transcription routes**\n were exercised with real payloads. Seventeen embedding and all five\n transcription routes passed. Cloudflare returned 429 for one embedding,\n GitHub returned 401 for two embeddings with the locally unavailable\n credential, and NVIDIA returned one isolated 500; these remain enabled\n because none is retirement evidence.\n\n## Current enabled-route result\n\nThe resulting catalog has **24 providers, 222 enabled chat routes, and 407\ncataloged chat models**.\n\n| Result for the 222 currently enabled chat routes | Routes |\n|---|---:|\n| Non-empty live completion | 126 |\n| Provider rate-limited during the audit (HTTP 429) | 31 |\n| GitHub maintenance credential rejected (HTTP 401) | 20 |\n| Hugging Face account exhausted included credits (HTTP 402) | 21 |\n| Isolated NVIDIA read timeout | 2 |\n| Not completion-tested because no safe usable configuration was available | 22 |\n\nThe non-success classifications above describe account or transient state, not\nmodel retirement, and therefore did not change catalog state.\n\n## Disabled after repeat definitive failure\n\nEach route below failed the full sweep and three independent repeat canaries\nwith the same definitive status:\n\n- Groq (404): `meta-llama/llama-4-scout-17b-16e-instruct`,\n `qwen/qwen3-32b`.\n- Hugging Face (400 unsupported): `CohereLabs/command-a-vision-07-2025`.\n- Kilo Gateway (404 unavailable/free period ended):\n `kwaipilot/kat-coder-pro-v2.5:free`, `poolside/laguna-m.1:free`,\n `tencent/hy3:free`.\n- LLM7 (400 unavailable): `minimax-m2.7`.\n- NVIDIA (410 Gone): `abacusai/dracarys-llama-3.1-70b-instruct`,\n `bytedance/seed-oss-36b-instruct`, `minimaxai/minimax-m2.7`,\n `mistralai/mistral-large-3-675b-instruct-2512`,\n `mistralai/mistral-small-4-119b-2603`,\n `qwen/qwen3-next-80b-a3b-instruct`, `qwen/qwen3.5-122b-a10b`,\n `sarvamai/sarvam-m`, `stepfun-ai/step-3.5-flash`, and\n `upstage/solar-10.7b-instruct`.\n- OpenRouter (404 no free endpoint):\n `cognitivecomputations/dolphin-mistral-24b-venice-edition:free`,\n `meta-llama/llama-3.2-3b-instruct:free`,\n `meta-llama/llama-3.3-70b-instruct:free`,\n `nousresearch/hermes-3-llama-3.1-405b:free`,\n `poolside/laguna-m.1:free`, `qwen/qwen3-coder:free`,\n `qwen/qwen3-next-80b-a3b-instruct:free`, and `tencent/hy3:free`.\n\nThe rows remain cataloged as disabled lifecycle history and can still be\nexplicitly pinned. They cannot enter automatic routing.\n\n## Gemini lifecycle\n\nGoogle's official deprecation table and June 1 release note mark\n`gemini-2.0-flash` and `gemini-2.0-flash-lite` as shut down on June 1, 2026.\nBoth remain cataloged as disabled lifecycle history.\n\nThe maintenance environment had no `GEMINI_API_KEY`, so no Gemini 3 route was\nenabled from documentation alone. `gemini-2.5-flash` remains enabled while\nGoogle lists its shutdown no earlier than October 16, 2026. A replacement must\npass repeat non-empty calls on a real free-tier project before enablement.\n\nOfficial sources:\n\n- <https://ai.google.dev/gemini-api/docs/deprecations>\n- <https://ai.google.dev/gemini-api/docs/changelog>\n- <https://ai.google.dev/gemini-api/docs/models>\n\n## LLM7 durable free selectors\n\nLLM7's official documentation recommends `default` and `fast`, states that\nconcrete model IDs are being phased out, and reserves `pro` for paid plans.\nAnonymous/keyless access is documented as free but rate-limited.\n\n| Selector | Successful HTTP 200 responses | Non-empty completions |\n|---|---:|---:|\n| `default` | 3/3 | 3/3 |\n| `fast` | 3/3 | 3/3 |\n\nThe `fast` canary used a 128-token output budget because its selected reasoning\nmodel consumed a smaller budget before emitting visible content. The catalog\nadds `default` and `fast` with conservative request hints and deliberately omits\nthe paid `pro` selector.\n\nOfficial sources:\n\n- <https://docs.llm7.io/guides/models>\n- <https://docs.llm7.io/quickstart>\n- <https://docs.llm7.io/limits>\n\n## Advisory discoveries and recoveries\n\nDiscovery reported many catalog gaps. They were not added because listing\npresence is not proof of durable free completion access. Four disabled OpenCode\nrows and one disabled NVIDIA row answered once, but one success does not meet\nthe three-success re-enable rule. OpenCode also remains subject to its existing\nprivacy/retention opt-in policy. No route was auto-enabled from discovery.\n"}}}} |