OpenCode
68a9071cf1
Local branch had diverged from origin/master (sibling commits on the same base). Rewrote local history linearly on top of origin/master, folding in all local content: exa/hot-platform discovery hunters, leaks ledger, nightly sweep orchestrator, updated .gitignore and skill docs, plus the latest hunt output and vault state. Remote-only files (vault keys, channel scripts) were restored rather than dropped, so the resulting tree is a full union of both sides.
1 line
36 KiB
JSON
1 line
36 KiB
JSON
{"repo": "abtreece/llm-bench", "files": {"docs/superpowers/plans/2026-06-12-hardware-aware-model-recommendation.md": {"8c62d6fb25213d8f6137e63900a6e885f2f5dfd0": {"t": 1786068570.2042687, "n": 34670, "h": "ff82e2578de28fa9", "c": "# Hardware-Aware Model Recommendation Implementation Plan\n\n> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.\n\n**Goal:** An advisory `python -m harness.recommend` command that probes the host's accelerator/RAM budget and reports which Ollama models fit, which installed ones don't, and which catalog models are worth pulling — plus a paste-ready `models.yaml` snippet.\n\n**Architecture:** A new `harness/recommend.py` module with a pure-function core (parsers for `nvidia-smi`/`rocm-smi`/`/proc/meminfo`/`sysctl`/`lspci`, a backend-dispatch function, fit rule, tier selection, renderers) and a thin subprocess/CLI shell. A curated `catalog.yaml` at repo root supplies \"worth pulling\" candidates. `harness/ollama_client.list_local_models` is repurposed to return names + sizes from `/api/tags`.\n\n**Tech Stack:** Python 3.12, stdlib (`dataclasses`, `subprocess`, `platform`, `argparse`, `json`, `re`), PyYAML, requests, pytest. Tests live in `harness_tests/` (outside the frozen `testpaths`), run explicitly.\n\n**Spec:** `docs/superpowers/specs/2026-06-12-hardware-aware-model-recommendation-design.md` (in git: `git show ce9b7fe:docs/superpowers/specs/2026-06-12-hardware-aware-model-recommendation-design.md`)\n\n**Registry-verified sizes (2026-06-12, ollama.com):** qwen2.5-coder:1.5b = 986MB ≈ 1.0 GB, :7b = 4.7, :14b = 9.0, :32b = 20.0, qwen3:8b = 5.2, qwen3:14b = 9.3, deepseek-coder-v2:16b = 8.9, qwen3-coder:30b = **19.0** (spec's 18.0 was a placeholder; corrected).\n\n---\n\n### Task 0: Workspace setup\n\n**Files:** none (environment only)\n\n- [x] **Step 1: Create venv and install deps** (worktree has no `.venv`)\n\n```bash\npython3 -m venv .venv\n.venv/bin/pip install -e .\n```\n\n- [x] **Step 2: Verify existing harness tests pass**\n\nRun: `.venv/bin/pytest harness_tests/ -q`\nExpected: all pass.\n\n---\n\n### Task 1: `ollama_client.list_local_models` returns names + sizes\n\n**Files:**\n- Modify: `harness/ollama_client.py:95-98`\n- Test: `harness_tests/test_recommend.py` (new file)\n\n`/api/tags` returns `{\"models\": [{\"name\": \"qwen2.5:14b\", \"size\": 8988124416, ...}]}` — `size` is bytes. Convert GB = bytes / 1e9 (matches the registry's decimal-GB convention and `models.yaml`). A pure `models_from_tags(data)` carries the logic so tests need no HTTP.\n\n- [x] **Step 1: Write failing tests**\n\nCreate `harness_tests/test_recommend.py`:\n\n```python\n\"\"\"Tests for harness/recommend.py — pure-function core, no subprocess/network.\"\"\"\nfrom harness import ollama_client\n\n\nclass TestModelsFromTags:\n def test_converts_bytes_to_decimal_gb(self):\n data = {\"models\": [{\"name\": \"qwen2.5:14b\", \"size\": 9_000_000_000}]}\n assert ollama_client.models_from_tags(data) == [\n {\"name\": \"qwen2.5:14b\", \"size_gb\": 9.0}\n ]\n\n def test_empty_and_missing_models_key(self):\n assert ollama_client.models_from_tags({}) == []\n assert ollama_client.models_from_tags({\"models\": []}) == []\n```\n\n- [x] **Step 2: Run to verify failure**\n\nRun: `.venv/bin/pytest harness_tests/test_recommend.py -q`\nExpected: FAIL — `AttributeError: ... no attribute 'models_from_tags'`\n\n- [x] **Step 3: Implement**\n\nIn `harness/ollama_client.py`, replace `list_local_models` (the `Iterable` import becomes unused — remove it from the `typing` import line):\n\n```python\ndef models_from_tags(data: dict) -> list[dict]:\n \"\"\"Pure extraction from an /api/tags payload: [{\"name\", \"size_gb\"}].\n\n Ollama reports size in bytes; GB = bytes / 1e9 to match the registry's\n decimal-GB convention used in models.yaml and catalog.yaml.\n \"\"\"\n return [\n {\"name\": m[\"name\"], \"size_gb\": int(m.get(\"size\", 0)) / 1e9}\n for m in data.get(\"models\", [])\n ]\n\n\ndef list_local_models(base_url: str = DEFAULT_BASE_URL) -> list[dict]:\n r = requests.get(f\"{base_url}/api/tags\", timeout=10)\n r.raise_for_status()\n return models_from_tags(r.json())\n```\n\n- [x] **Step 4: Run tests** — `.venv/bin/pytest harness_tests/test_recommend.py -q` → PASS; full `.venv/bin/pytest harness_tests/ -q` still green.\n\n- [x] **Step 5: Commit** — `git add harness/ollama_client.py harness_tests/test_recommend.py && git commit -m \"feat: return name and size_gb from list_local_models\"`\n\n---\n\n### Task 2: `catalog.yaml` + loader\n\n**Files:**\n- Create: `catalog.yaml` (repo root)\n- Create: `harness/recommend.py`\n- Test: `harness_tests/test_recommend.py`\n\n- [x] **Step 1: Write failing tests** (append to `harness_tests/test_recommend.py`)\n\n```python\nfrom harness import recommend\n\n\nclass TestCatalog:\n def test_repo_catalog_loads_and_is_well_formed(self):\n entries = recommend.load_catalog()\n assert len(entries) >= 8\n for e in entries:\n assert isinstance(e[\"name\"], str) and \":\" in e[\"name\"]\n assert e[\"size_gb\"] > 0\n\n def test_missing_catalog_degrades_to_empty(self, tmp_path):\n assert recommend.load_catalog(tmp_path / \"nope.yaml\") == []\n```\n\n- [x] **Step 2: Run to verify failure** — `ModuleNotFoundError`/`ImportError` on `recommend`.\n\n- [x] **Step 3: Create `catalog.yaml`** (registry-verified sizes, smallest first):\n\n```yaml\n# Curated coder-capable models for harness.recommend suggestions.\n# size_gb = on-disk size of the default (q4) tag, per the Ollama registry\n# (verified 2026-06-12). Spanning 1.5B-32B so every hardware tier gets\n# at least one suggestion.\ncatalog:\n - name: qwen2.5-coder:1.5b\n size_gb: 1.0\n notes: floor sanity check\n - name: qwen2.5-coder:7b\n size_gb: 4.7\n - name: deepseek-coder-v2:16b\n size_gb: 8.9\n - name: qwen2.5-coder:14b\n size_gb: 9.0\n - name: qwen3:8b\n size_gb: 5.2\n - name: qwen3:14b\n size_gb: 9.3\n - name: qwen3-coder:30b\n size_gb: 19.0\n - name: qwen2.5-coder:32b\n size_gb: 20.0\n```\n\n(Ordering nicety only; selection sorts explicitly.)\n\n- [x] **Step 4: Create `harness/recommend.py`** with module header + loader:\n\n```python\n\"\"\"Hardware-aware model recommendation (advisory only).\n\nProbes the host's accelerator/RAM budget, checks installed Ollama models\nand the curated catalog.yaml against it, and prints a report plus a\npaste-ready models.yaml snippet. Never writes any file; models.yaml\nremains the single source of truth for harness.run.\n\nCLI:\n python -m harness.recommend [--headroom-gb 2.0]\n\"\"\"\nfrom __future__ import annotations\n\nfrom pathlib import Path\n\nimport yaml\n\nREPO = Path(__file__).resolve().parents[1]\nCATALOG_YAML = REPO / \"catalog.yaml\"\n\n\ndef load_catalog(path: Path = CATALOG_YAML) -> list[dict]:\n # Missing file is fine (no suggestions); a malformed one should surface\n # loudly rather than silently dropping the tier (same philosophy as\n # report.py's models.yaml handling).\n try:\n data = yaml.safe_load(path.read_text())\n except FileNotFoundError:\n return []\n return [\n {\"name\": str(e[\"name\"]), \"size_gb\": float(e[\"size_gb\"])}\n for e in data[\"catalog\"]\n ]\n```\n\n- [x] **Step 5: Run tests** → PASS. **Commit:** `git add catalog.yaml harness/recommend.py harness_tests/test_recommend.py && git commit -m \"feat: add curated model catalog and loader\"`\n\n---\n\n### Task 3: Probe-output parsers (pure functions)\n\n**Files:**\n- Modify: `harness/recommend.py`\n- Test: `harness_tests/test_recommend.py`\n\nUnit normalization: nvidia-smi `--nounits` memory.total is **MiB** → GB = MiB·1024²/1e9. `/proc/meminfo` MemTotal is **KiB** → GB = KiB·1024/1e9. `sysctl hw.memsize` is **bytes** → /1e9. rocm-smi JSON `\"VRAM Total Memory (B)\"` is **bytes** → /1e9.\n\n- [x] **Step 1: Write failing tests**\n\n```python\nNVIDIA_ONE = \"Tesla T4, 15360\\n\"\nNVIDIA_TWO = \"Tesla T4, 15360\\nTesla T4, 15360\\n\"\nROCM_JSON = '{\"card0\": {\"VRAM Total Memory (B)\": \"17163091968\", \"VRAM Total Used Memory (B)\": \"0\"}}'\nMEMINFO = \"MemTotal: 65856380 kB\\nMemFree: 1234 kB\\n\"\nLSPCI_NVIDIA = \"01:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)\\n\"\nLSPCI_AMD = \"03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21\\n\"\nLSPCI_NONE = \"00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 630\\n\"\n\n\nclass TestParsers:\n def test_nvidia_smi_single_gpu(self):\n gpus = recommend.parse_nvidia_smi(NVIDIA_ONE)\n assert len(gpus) == 1\n assert gpus[0].name == \"Tesla T4\"\n assert abs(gpus[0].vram_gb - 16.1) < 0.1 # 15360 MiB\n\n def test_nvidia_smi_multi_gpu(self):\n gpus = recommend.parse_nvidia_smi(NVIDIA_TWO)\n assert len(gpus) == 2\n assert abs(sum(g.vram_gb for g in gpus) - 32.2) < 0.2\n\n def test_nvidia_smi_garbage(self):\n assert recommend.parse_nvidia_smi(\"NVIDIA-SMI has failed\\n\") == []\n assert recommend.parse_nvidia_smi(\"\") == []\n\n def test_rocm_smi_happy_path(self):\n gpus = recommend.parse_rocm_smi(ROCM_JSON)\n assert len(gpus) == 1\n assert abs(gpus[0].vram_gb - 17.2) < 0.1\n\n def test_rocm_smi_missing_key(self):\n assert recommend.parse_rocm_smi('{\"card0\": {\"other\": \"1\"}}') == []\n assert recommend.parse_rocm_smi(\"not json\") == []\n\n def test_meminfo(self):\n assert abs(recommend.parse_meminfo(MEMINFO) - 67.4) < 0.1\n\n def test_meminfo_garbage(self):\n assert recommend.parse_meminfo(\"nope\") is None\n\n def test_sysctl_memsize(self):\n assert recommend.parse_sysctl_memsize(\"68719476736\\n\") == 68.719476736\n\n def test_sysctl_garbage(self):\n assert recommend.parse_sysctl_memsize(\"zzz\") is None\n\n class TestLspciScan:\n def test_nvidia_hit(self):\n assert recommend.scan_lspci(LSPCI_NVIDIA) == \"nvidia\"\n\n def test_amd_hit(self):\n assert recommend.scan_lspci(LSPCI_AMD) == \"amd\"\n\n def test_no_discrete_gpu(self):\n assert recommend.scan_lspci(LSPCI_NONE) is None\n\n def test_empty(self):\n assert recommend.scan_lspci(\"\") is None\n```\n\n- [x] **Step 2: Run to verify failure.**\n\n- [x] **Step 3: Implement** (add to `harness/recommend.py`; `import json`, `import re`, `from dataclasses import dataclass, field`):\n\n```python\n@dataclass(frozen=True)\nclass Gpu:\n name: str\n vram_gb: float\n\n\ndef parse_nvidia_smi(text: str) -> list[Gpu]:\n \"\"\"Parse `nvidia-smi --query-gpu=name,memory.total --format=csv,noheader,nounits`.\n\n memory.total is MiB with --nounits.\n \"\"\"\n gpus: list[Gpu] = []\n for line in text.splitlines():\n name, sep, mib = line.rpartition(\",\")\n if not sep:\n continue\n try:\n vram_gb = float(mib.strip()) * 1024**2 / 1e9\n except ValueError:\n continue\n gpus.append(Gpu(name=name.strip(), vram_gb=vram_gb))\n return gpus\n\n\ndef parse_rocm_smi(text: str) -> list[Gpu]:\n \"\"\"Parse `rocm-smi --showmeminfo vram --json` (best-effort: pinned to the\n documented {\"cardN\": {\"VRAM Total Memory (B)\": \"<bytes>\"}} shape; not\n verified on real ROCm hardware).\"\"\"\n try:\n data = json.loads(text)\n except json.JSONDecodeError:\n return []\n gpus: list[Gpu] = []\n for card, info in sorted(data.items()):\n if not isinstance(info, dict):\n continue\n raw = info.get(\"VRAM Total Memory (B)\")\n if raw is None:\n continue\n try:\n gpus.append(Gpu(name=card, vram_gb=float(raw) / 1e9))\n except ValueError:\n continue\n return gpus\n\n\ndef parse_meminfo(text: str) -> float | None:\n m = re.search(r\"^MemTotal:\\s+(\\d+)\\s*kB\", text, re.MULTILINE)\n if not m:\n return None\n return int(m.group(1)) * 1024 / 1e9 # meminfo kB is KiB\n\n\ndef parse_sysctl_memsize(text: str) -> float | None:\n try:\n return int(text.strip()) / 1e9\n except ValueError:\n return None\n\n\ndef scan_lspci(text: str) -> str | None:\n \"\"\"Spot a discrete NVIDIA/AMD VGA/3D controller a vendor tool missed.\"\"\"\n for line in text.splitlines():\n if \"VGA compatible controller\" not in line and \"3D controller\" not in line:\n continue\n if \"NVIDIA\" in line:\n return \"nvidia\"\n if \"AMD\" in line or \"ATI\" in line:\n return \"amd\"\n return None\n```\n\n- [x] **Step 4: Run tests** → PASS. **Commit:** `git commit -am \"feat: add hardware probe-output parsers\"`\n\n---\n\n### Task 4: Backend dispatch + budget (pure `build_hardware`)\n\n**Files:**\n- Modify: `harness/recommend.py`\n- Test: `harness_tests/test_recommend.py`\n\n`build_hardware` takes raw probe results (`str | None` per tool; `None` = tool missing/failed) and makes every decision; the subprocess shell comes in Task 7. Budget rule: cuda/rocm = Σ VRAM; metal = 0.67·RAM; cpu = RAM − 4.0.\n\n- [x] **Step 1: Write failing tests**\n\n```python\nclass TestBuildHardware:\n def test_darwin_arm64_is_metal_unified_fraction(self):\n hw = recommend.build_hardware(\n \"Darwin\", \"arm64\", nvidia_out=None, rocm_out=None,\n lspci_out=None, ram_gb=64.0)\n assert hw.backend == \"metal\"\n assert hw.gpus == []\n assert abs(hw.budget_gb - 0.67 * 64.0) < 0.01\n\n def test_darwin_x86_is_cpu(self):\n hw = recommend.build_hardware(\n \"Darwin\", \"x86_64\", nvidia_out=None, rocm_out=None,\n lspci_out=None, ram_gb=32.0)\n assert hw.backend == \"cpu\"\n assert hw.budget_gb == 32.0 - 4.0\n\n def test_linux_nvidia(self):\n hw = recommend.build_hardware(\n \"Linux\", \"x86_64\", nvidia_out=NVIDIA_ONE, rocm_out=None,\n lspci_out=None, ram_gb=64.0)\n assert hw.backend == \"cuda\"\n assert abs(hw.budget_gb - 16.1) < 0.1\n\n def test_linux_nvidia_multi_gpu_sums(self):\n hw = recommend.build_hardware(\n \"Linux\", \"x86_64\", nvidia_out=NVIDIA_TWO, rocm_out=None,\n lspci_out=None, ram_gb=64.0)\n assert abs(hw.budget_gb - 32.2) < 0.2\n\n def test_linux_rocm(self):\n hw = recommend.build_hardware(\n \"Linux\", \"x86_64\", nvidia_out=None, rocm_out=ROCM_JSON,\n lspci_out=None, ram_gb=64.0)\n assert hw.backend == \"rocm\"\n assert abs(hw.budget_gb - 17.2) < 0.1\n\n def test_linux_nvidia_smi_garbage_falls_through(self):\n # nvidia-smi present but emitting garbage = driver gap, not cuda.\n hw = recommend.build_hardware(\n \"Linux\", \"x86_64\", nvidia_out=\"NVIDIA-SMI has failed\\n\",\n rocm_out=None, lspci_out=LSPCI_NVIDIA, ram_gb=64.0)\n assert hw.backend == \"cpu\"\n assert any(\"NVIDIA\" in w for w in hw.warnings)\n\n def test_linux_driver_gap_warns_and_uses_cpu(self):\n hw = recommend.build_hardware(\n \"Linux\", \"x86_64\", nvidia_out=None, rocm_out=None,\n lspci_out=LSPCI_NVIDIA, ram_gb=64.0)\n assert hw.backend == \"cpu\"\n assert hw.budget_gb == 60.0\n assert any(\"NVIDIA\" in w for w in hw.warnings)\n\n def test_linux_amd_driver_gap_warns(self):\n hw = recommend.build_hardware(\n \"Linux\", \"x86_64\", nvidia_out=None, rocm_out=None,\n lspci_out=LSPCI_AMD, ram_gb=64.0)\n assert hw.backend == \"cpu\"\n assert any(\"ROCm\" in w for w in hw.warnings)\n\n def test_linux_no_lspci_notes_incomplete_detection(self):\n hw = recommend.build_hardware(\n \"Linux\", \"x86_64\", nvidia_out=None, rocm_out=None,\n lspci_out=None, ram_gb=64.0)\n assert hw.backend == \"cpu\"\n assert any(\"lspci\" in w for w in hw.warnings)\n\n def test_unsupported_os_raises(self):\n import pytest\n with pytest.raises(recommend.RecommendError):\n recommend.build_hardware(\"Windows\", \"AMD64\", nvidia_out=None,\n rocm_out=None, lspci_out=None, ram_gb=8.0)\n\n def test_no_gpu_and_no_ram_raises(self):\n import pytest\n with pytest.raises(recommend.RecommendError):\n recommend.build_hardware(\"Linux\", \"x86_64\", nvidia_out=None,\n rocm_out=None, lspci_out=\"\", ram_gb=None)\n```\n\n- [x] **Step 2: Run to verify failure.**\n\n- [x] **Step 3: Implement**\n\n```python\n# Metal's default wired-memory limit gives the GPU roughly two-thirds of\n# physical RAM on Apple Silicon; --headroom-gb is the escape hatch.\nAPPLE_UNIFIED_FRACTION = 0.67\n# RAM kept back for the OS and Ollama runtime on CPU-only hosts.\nCPU_OS_RESERVE_GB = 4.0\n\n\nclass RecommendError(Exception):\n \"\"\"Nothing to fit against (unsupported OS, no GPU and no readable RAM).\"\"\"\n\n\n@dataclass(frozen=True)\nclass Hardware:\n backend: str # \"cuda\" | \"rocm\" | \"metal\" | \"cpu\"\n gpus: list[Gpu]\n ram_gb: float | None\n budget_gb: float\n warnings: list[str] = field(default_factory=list)\n\n\ndef build_hardware(\n system: str,\n machine: str,\n *,\n nvidia_out: str | None,\n rocm_out: str | None,\n lspci_out: str | None,\n ram_gb: float | None,\n) -> Hardware:\n \"\"\"Decide backend and memory budget from raw probe output.\n\n Pure: probe_hardware() gathers the raw text, this makes every decision.\n \"\"\"\n warnings: list[str] = []\n\n if system == \"Darwin\":\n if ram_gb is None:\n raise RecommendError(\"could not read RAM size via sysctl\")\n if machine == \"arm64\":\n return Hardware(\"metal\", [], ram_gb,\n APPLE_UNIFIED_FRACTION * ram_gb, warnings)\n warnings.append(\n \"Intel Mac: Ollama does not accelerate Intel GPUs; using CPU budget.\")\n return Hardware(\"cpu\", [], ram_gb, ram_gb - CPU_OS_RESERVE_GB, warnings)\n\n if system != \"Linux\":\n raise RecommendError(f\"unsupported OS: {system}\")\n\n gpus = parse_nvidia_smi(nvidia_out) if nvidia_out is not None else []\n if gpus:\n return Hardware(\"cuda\", gpus, ram_gb,\n sum(g.vram_gb for g in gpus), warnings)\n\n gpus = parse_rocm_smi(rocm_out) if rocm_out is not None else []\n if gpus:\n return Hardware(\"rocm\", gpus, ram_gb,\n sum(g.vram_gb for g in gpus), warnings)\n\n if lspci_out is None:\n warnings.append(\n \"GPU detection incomplete: lspci not available (pciutils); \"\n \"assuming CPU-only.\")\n else:\n vendor = scan_lspci(lspci_out)\n if vendor == \"nvidia\":\n warnings.append(\n \"lspci shows an NVIDIA GPU but nvidia-smi is missing or \"\n \"unusable — install the NVIDIA driver to use it. \"\n \"Falling back to CPU budget.\")\n elif vendor == \"amd\":\n warnings.append(\n \"lspci shows an AMD GPU but rocm-smi is missing or \"\n \"unusable — install ROCm to use it. \"\n \"Falling back to CPU budget.\")\n\n if ram_gb is None:\n raise RecommendError(\"no usable GPU and could not read RAM size\")\n return Hardware(\"cpu\", [], ram_gb, ram_gb - CPU_OS_RESERVE_GB, warnings)\n```\n\n- [x] **Step 4: Run tests** → PASS. **Commit:** `git commit -am \"feat: add backend dispatch and memory budget\"`\n\n---\n\n### Task 5: Fit rule + tier selection\n\n**Files:**\n- Modify: `harness/recommend.py`\n- Test: `harness_tests/test_recommend.py`\n\n`fits(model) := size_gb + headroom_gb <= budget_gb`. Tiers: selected (installed ∧ fits), excluded (installed ∧ ¬fits), worth pulling (catalog ∧ fits ∧ ¬installed, exact tag match).\n\n- [x] **Step 1: Write failing tests**\n\n```python\nT4_BUDGET = 16.1 # 15360 MiB\n\n\nclass TestFitRule:\n def test_t4_anchor_30b_excluded(self):\n assert recommend.fits(19.0, T4_BUDGET, 2.0) is False\n\n def test_t4_anchor_14b_admitted(self):\n assert recommend.fits(9.0, T4_BUDGET, 2.0) is True\n\n def test_boundary_is_inclusive(self):\n assert recommend.fits(14.0, 16.0, 2.0) is True\n assert recommend.fits(14.1, 16.0, 2.0) is False\n\n def test_headroom_override(self):\n assert recommend.fits(15.5, T4_BUDGET, 0.5) is True\n assert recommend.fits(15.5, T4_BUDGET, 2.0) is False\n\n\nclass TestTierSelection:\n INSTALLED = [\n {\"name\": \"qwen2.5:14b\", \"size_gb\": 9.0},\n {\"name\": \"qwen3-coder:30b\", \"size_gb\": 19.0},\n ]\n CATALOG = [\n {\"name\": \"qwen2.5-coder:14b\", \"size_gb\": 9.0},\n {\"name\": \"qwen2.5-coder:32b\", \"size_gb\": 20.0},\n {\"name\": \"qwen2.5:14b\", \"size_gb\": 9.0}, # also installed\n ]\n\n def test_tiers_on_t4(self):\n tiers = recommend.select_tiers(\n self.INSTALLED, self.CATALOG, budget_gb=T4_BUDGET, headroom_gb=2.0)\n assert [m.name for m in tiers.selected] == [\"qwen2.5:14b\"]\n assert [m.name for m in tiers.excluded] == [\"qwen3-coder:30b\"]\n # 32b doesn't fit, qwen2.5:14b already installed\n assert [m.name for m in tiers.worth_pulling] == [\"qwen2.5-coder:14b\"]\n\n def test_margin_is_budget_minus_size_minus_headroom(self):\n tiers = recommend.select_tiers(\n self.INSTALLED, self.CATALOG, budget_gb=T4_BUDGET, headroom_gb=2.0)\n assert abs(tiers.selected[0].margin_gb - (T4_BUDGET - 9.0 - 2.0)) < 0.01\n\n def test_exact_tag_match_only(self):\n installed = [{\"name\": \"qwen2.5-coder:14b-instruct-q8_0\", \"size_gb\": 15.0}]\n catalog = [{\"name\": \"qwen2.5-coder:14b\", \"size_gb\": 9.0}]\n tiers = recommend.select_tiers(installed, catalog,\n budget_gb=20.0, headroom_gb=2.0)\n assert [m.name for m in tiers.worth_pulling] == [\"qwen2.5-coder:14b\"]\n\n def test_empty_installed_still_suggests_catalog(self):\n tiers = recommend.select_tiers([], self.CATALOG,\n budget_gb=T4_BUDGET, headroom_gb=2.0)\n assert tiers.selected == [] and tiers.excluded == []\n assert len(tiers.worth_pulling) == 2\n\n def test_tiers_sorted_smallest_first(self):\n installed = [\n {\"name\": \"big\", \"size_gb\": 9.0},\n {\"name\": \"small\", \"size_gb\": 1.0},\n ]\n tiers = recommend.select_tiers(installed, [],\n budget_gb=16.0, headroom_gb=2.0)\n assert [m.name for m in tiers.selected] == [\"small\", \"big\"]\n```\n\n- [x] **Step 2: Run to verify failure.**\n\n- [x] **Step 3: Implement**\n\n```python\nDEFAULT_HEADROOM_GB = 2.0 # KV cache at num_ctx=16384 plus runtime overhead\n\n\ndef fits(size_gb: float, budget_gb: float, headroom_gb: float) -> bool:\n return size_gb + headroom_gb <= budget_gb\n\n\n@dataclass(frozen=True)\nclass ModelFit:\n name: str\n size_gb: float\n margin_gb: float # budget - size - headroom; negative when excluded\n\n\n@dataclass(frozen=True)\nclass Tiers:\n selected: list[ModelFit] # installed, fits\n excluded: list[ModelFit] # installed, does not fit\n worth_pulling: list[ModelFit] # catalog, fits, not installed\n\n\ndef select_tiers(\n installed: list[dict],\n catalog: list[dict],\n *,\n budget_gb: float,\n headroom_gb: float,\n) -> Tiers:\n def fit(m: dict) -> ModelFit:\n return ModelFit(m[\"name\"], m[\"size_gb\"],\n budget_gb - m[\"size_gb\"] - headroom_gb)\n\n by_size = lambda f: f.size_gb\n installed_names = {m[\"name\"] for m in installed}\n selected = sorted(\n (fit(m) for m in installed if fits(m[\"size_gb\"], budget_gb, headroom_gb)),\n key=by_size)\n excluded = sorted(\n (fit(m) for m in installed if not fits(m[\"size_gb\"], budget_gb, headroom_gb)),\n key=by_size)\n worth_pulling = sorted(\n (fit(m) for m in catalog\n if m[\"name\"] not in installed_names\n and fits(m[\"size_gb\"], budget_gb, headroom_gb)),\n key=by_size)\n return Tiers(selected, excluded, worth_pulling)\n```\n\n- [x] **Step 4: Run tests** → PASS. **Commit:** `git commit -am \"feat: add fit rule and selection tiers\"`\n\n---\n\n### Task 6: Report + snippet rendering\n\n**Files:**\n- Modify: `harness/recommend.py`\n- Test: `harness_tests/test_recommend.py`\n\n- [x] **Step 1: Write failing tests**\n\n```python\nimport yaml as yaml_mod\n\n\ndef make_hw(**kw):\n base = dict(backend=\"cuda\",\n gpus=[recommend.Gpu(\"Tesla T4\", 16.1)],\n ram_gb=67.4, budget_gb=16.1, warnings=[])\n base.update(kw)\n return recommend.Hardware(**base)\n\n\ndef make_tiers(selected=(), excluded=(), worth=()):\n return recommend.Tiers(list(selected), list(excluded), list(worth))\n\n\nclass TestRender:\n def test_report_sections_and_numbers(self):\n tiers = make_tiers(\n selected=[recommend.ModelFit(\"qwen2.5:14b\", 9.0, 5.1)],\n excluded=[recommend.ModelFit(\"qwen3-coder:30b\", 19.0, -4.9)],\n worth=[recommend.ModelFit(\"qwen2.5-coder:14b\", 9.0, 5.1)],\n )\n text = recommend.render_report(make_hw(), tiers, headroom_gb=2.0)\n assert \"backend: cuda\" in text\n assert \"Tesla T4 (16.1 GB VRAM)\" in text\n assert \"budget: 16.1 GB (headroom 2.0 GB)\" in text\n assert \"margin 5.1 GB\" in text\n assert \"needs 21.0 GB > 16.1 GB budget\" in text\n assert \"ollama pull qwen2.5-coder:14b\" in text\n\n def test_warnings_render(self):\n text = recommend.render_report(\n make_hw(warnings=[\"driver gap\"]), make_tiers(), headroom_gb=2.0)\n assert \"warning: driver gap\" in text\n\n def test_ollama_down_note_replaces_installed_tiers(self):\n text = recommend.render_report(\n make_hw(), make_tiers(), headroom_gb=2.0,\n ollama_note=\"Ollama not reachable at http://localhost:11434\")\n assert \"Ollama not reachable\" in text\n assert \"# selected\" not in text and \"# excluded\" not in text\n\n def test_empty_tiers_render_none(self):\n text = recommend.render_report(make_hw(), make_tiers(), headroom_gb=2.0)\n assert text.count(\"(none)\") >= 3\n\n def test_snippet_smallest_first_and_yaml_parseable(self):\n tiers = make_tiers(\n selected=[recommend.ModelFit(\"qwen2.5:14b\", 9.0, 5.1)],\n worth=[recommend.ModelFit(\"qwen2.5-coder:1.5b\", 1.0, 13.1)],\n )\n snippet = recommend.render_snippet(tiers)\n data = yaml_mod.safe_load(snippet)\n assert data == {\"models\": [\n {\"name\": \"qwen2.5-coder:1.5b\", \"size_gb\": 1.0},\n {\"name\": \"qwen2.5:14b\", \"size_gb\": 9.0},\n ]}\n\n def test_empty_snippet_is_omitted_from_report(self):\n text = recommend.render_report(make_hw(), make_tiers(), headroom_gb=2.0)\n assert \"models.yaml snippet\" not in text\n```\n\n- [x] **Step 2: Run to verify failure.**\n\n- [x] **Step 3: Implement**\n\n```python\ndef render_snippet(tiers: Tiers) -> str:\n \"\"\"models.yaml-shaped YAML of selected + worth-pulling, smallest first.\"\"\"\n models = sorted(tiers.selected + tiers.worth_pulling,\n key=lambda f: f.size_gb)\n lines = [\"models:\"]\n for f in models:\n lines.append(f\" - name: {f.name}\")\n lines.append(f\" size_gb: {f.size_gb:.1f}\")\n return \"\\n\".join(lines) + \"\\n\"\n\n\ndef render_report(\n hw: Hardware,\n tiers: Tiers,\n *,\n headroom_gb: float,\n ollama_note: str | None = None,\n) -> str:\n out: list[str] = []\n out.append(\"# hardware\")\n out.append(f\"backend: {hw.backend}\")\n for g in hw.gpus:\n out.append(f\"gpu: {g.name} ({g.vram_gb:.1f} GB VRAM)\")\n if hw.ram_gb is not None:\n out.append(f\"ram: {hw.ram_gb:.1f} GB\")\n out.append(f\"budget: {hw.budget_gb:.1f} GB (headroom {headroom_gb:.1f} GB)\")\n for w in hw.warnings:\n out.append(f\"warning: {w}\")\n out.append(\"\")\n\n if ollama_note:\n out.append(f\"note: {ollama_note}\")\n out.append(\"\")\n else:\n out.append(\"# selected (installed, fits)\")\n for f in tiers.selected:\n out.append(f\"{f.name:<24} {f.size_gb:>5.1f} GB margin {f.margin_gb:.1f} GB\")\n if not tiers.selected:\n out.append(\"(none)\")\n out.append(\"\")\n\n out.append(\"# excluded (installed, does not fit)\")\n for f in tiers.excluded:\n out.append(\n f\"{f.name:<24} {f.size_gb:>5.1f} GB needs \"\n f\"{f.size_gb + headroom_gb:.1f} GB > {hw.budget_gb:.1f} GB budget\")\n if not tiers.excluded:\n out.append(\"(none)\")\n out.append(\"\")\n\n out.append(\"# worth pulling (catalog, fits, not installed)\")\n for f in tiers.worth_pulling:\n out.append(f\"{f.name:<24} {f.size_gb:>5.1f} GB ollama pull {f.name}\")\n if not tiers.worth_pulling:\n out.append(\"(none)\")\n out.append(\"\")\n\n if tiers.selected or tiers.worth_pulling:\n out.append(\"# models.yaml snippet\")\n out.append(render_snippet(tiers))\n return \"\\n\".join(out)\n```\n\n- [x] **Step 4: Run tests** → PASS. **Commit:** `git commit -am \"feat: render recommendation report and models.yaml snippet\"`\n\n---\n\n### Task 7: Subprocess shell + CLI `main`\n\n**Files:**\n- Modify: `harness/recommend.py`\n- Test: `harness_tests/test_recommend.py` (main wiring only; no subprocess in tests)\n\n- [x] **Step 1: Write failing test** (exit codes via monkeypatched probe — still no subprocess):\n\n```python\nclass TestMain:\n def test_unsupported_os_exits_nonzero(self, monkeypatch, capsys):\n monkeypatch.setattr(recommend.platform, \"system\", lambda: \"OpenVMS\")\n rc = recommend.main([])\n assert rc != 0\n assert \"unsupported OS\" in capsys.readouterr().err\n\n def test_ollama_down_still_reports(self, monkeypatch, capsys):\n monkeypatch.setattr(\n recommend, \"probe_hardware\",\n lambda: recommend.Hardware(\"cuda\", [recommend.Gpu(\"T4\", 16.1)],\n 64.0, 16.1, []))\n def boom(base_url=None):\n raise recommend.requests.ConnectionError(\"down\")\n monkeypatch.setattr(recommend.ollama_client, \"list_local_models\", boom)\n rc = recommend.main([])\n out = capsys.readouterr().out\n assert rc == 0\n assert \"Ollama not reachable\" in out\n assert \"worth pulling\" in out\n```\n\n- [x] **Step 2: Run to verify failure.**\n\n- [x] **Step 3: Implement** (add `import argparse`, `import platform`, `import subprocess`, `import sys`, `import requests`, `from harness import ollama_client`):\n\n```python\ndef _run(cmd: list[str]) -> str | None:\n \"\"\"Thin shell: command stdout on success, None if missing/broken.\"\"\"\n try:\n proc = subprocess.run(cmd, capture_output=True, text=True, timeout=10)\n except (OSError, subprocess.TimeoutExpired):\n return None\n if proc.returncode != 0:\n return None\n return proc.stdout\n\n\ndef probe_hardware() -> Hardware:\n system = platform.system()\n machine = platform.machine()\n\n if system == \"Darwin\":\n out = _run([\"sysctl\", \"-n\", \"hw.memsize\"])\n ram_gb = parse_sysctl_memsize(out) if out is not None else None\n return build_hardware(system, machine, nvidia_out=None,\n rocm_out=None, lspci_out=None, ram_gb=ram_gb)\n\n if system == \"Linux\":\n try:\n ram_gb = parse_meminfo(Path(\"/proc/meminfo\").read_text())\n except OSError:\n ram_gb = None\n nvidia_out = _run([\"nvidia-smi\", \"--query-gpu=name,memory.total\",\n \"--format=csv,noheader,nounits\"])\n # Only shell out to the next tool when the previous one yielded nothing.\n rocm_out = None\n lspci_out = None\n if not (nvidia_out and parse_nvidia_smi(nvidia_out)):\n rocm_out = _run([\"rocm-smi\", \"--showmeminfo\", \"vram\", \"--json\"])\n if not (rocm_out and parse_rocm_smi(rocm_out)):\n lspci_out = _run([\"lspci\"])\n return build_hardware(system, machine, nvidia_out=nvidia_out,\n rocm_out=rocm_out, lspci_out=lspci_out,\n ram_gb=ram_gb)\n\n return build_hardware(system, machine, nvidia_out=None, rocm_out=None,\n lspci_out=None, ram_gb=None)\n\n\ndef main(argv: list[str] | None = None) -> int:\n p = argparse.ArgumentParser(\n description=\"Recommend Ollama models for this host (advisory; \"\n \"never writes models.yaml).\")\n p.add_argument(\"--headroom-gb\", type=float, default=DEFAULT_HEADROOM_GB,\n help=\"memory kept free for KV cache and runtime overhead \"\n f\"(default {DEFAULT_HEADROOM_GB})\")\n args = p.parse_args(argv)\n\n try:\n hw = probe_hardware()\n except RecommendError as e:\n print(f\"error: {e}\", file=sys.stderr)\n return 1\n\n ollama_note = None\n installed: list[dict] = []\n try:\n installed = ollama_client.list_local_models()\n except requests.RequestException:\n ollama_note = (\n f\"Ollama not reachable at {ollama_client.DEFAULT_BASE_URL} — \"\n \"installed-model tiers skipped.\")\n\n tiers = select_tiers(installed, load_catalog(),\n budget_gb=hw.budget_gb, headroom_gb=args.headroom_gb)\n sys.stdout.write(render_report(hw, tiers, headroom_gb=args.headroom_gb,\n ollama_note=ollama_note))\n return 0\n\n\nif __name__ == \"__main__\":\n sys.exit(main())\n```\n\nNote: `test_unsupported_os_exits_nonzero` monkeypatches `platform.system`, so `probe_hardware` must reach `build_hardware` for the non-Darwin/non-Linux case (it does — last return) and `main` converts the `RecommendError` to exit 1.\n\n- [x] **Step 4: Run tests** → PASS.\n\n- [x] **Step 5: Smoke-run the CLI on this host (Darwin/arm64 → metal path)**\n\nRun: `.venv/bin/python -m harness.recommend`\nExpected: hardware section with `backend: metal`, a budget ≈ 0.67·RAM, an \"Ollama not reachable\" note (or installed tiers if a local daemon is up), and catalog suggestions. Exit 0.\n\n- [x] **Step 6: Commit** — `git commit -am \"feat: add harness.recommend CLI with hardware probe\"`\n\n---\n\n### Task 8: Full verification\n\n- [x] **Step 1: Full test suite** — `.venv/bin/pytest harness_tests/ tests/ -q` → all green.\n- [x] **Step 2: Re-read spec, confirm every requirement maps to shipped code** (probe table, fit rule + constants, tiers, output sketch, error handling rows, test list).\n- [x] **Step 3: Commit plan doc** — `git add docs/superpowers/plans/ && git commit -m \"docs: implementation plan for hardware-aware model recommendation\"`\n\n---\n\n## Self-review notes\n\n- **Spec coverage:** probe table → Tasks 3/4/7; fit rule + constants → Tasks 4/5; tiers → Task 5; output sketch + snippet → Task 6; catalog → Task 2; ollama_client change → Task 1; all four error-handling rows → Tasks 4 (RecommendError) and 7 (main); every test bullet in the spec has a named test class.\n- **Spec deviation (deliberate):** `qwen3-coder:30b` size corrected 18.0 → 19.0 per registry verification; the T4 sanity anchor still holds (19+2=21 > 16.1).\n- **Open item resolved:** catalog sizes verified against ollama.com on 2026-06-12. rocm-smi parser marked best-effort in its docstring, pinned to documented output shape, per spec.\n"}}}} |