1 line
20 KiB
JSON
1 line
20 KiB
JSON
{"repo": "atomic-germ/guanaco", "files": {"README.md": {"d1f7488765c3876af5055f1dce76f0271f2d2627": {"t": 1785747254.1689641, "n": 6935, "h": "6ca805a0da936cd7", "c": "# Guanaco (llama.cpp delta)\n\nGuanaco is a disk-streaming add-on for **llama.cpp MoE models**. It assumes you\nalready know how to build and run llama.cpp; this document only covers what is\n*different*.\n\nThe one idea: a Mixture-of-Experts model routes each token to a handful of\nexperts out of many, so most expert weights are idle at any moment. Plain\nllama.cpp memory-maps the whole GGUF and relies on the OS page cache to keep\nthe working set in RAM. Guanaco instead **keeps only a small set of \"hot\"\nexperts pinned in RAM and streams the rest from NVMe on demand**, so a model\nthat would never fit in system memory runs anyway — bounded by a knob you\ncontrol, not by the file size.\n\n<img width=\"2880\" height=\"1854\" alt=\"image\" src=\"https://github.com/user-attachments/assets/65311726-6a53-4cb8-abcc-5725d447269d\" />\nTest system is a Framework 13 AMD Ryzen 340 AI with 48GB of total ram/vram shared, 6 cores on the cpu. OS is Fedora 45 Rawhide, kernel 7.2. Both models; Qwen3.5 122B A10B and Qwen3.6 35B A3B are running concurrently on that system. Ram use is largely context, and the gpu is unused for now.\n\nDense (non-MoE) models are untouched currently (maybe forever: Guanaco detects no expert tensors and\ndoes nothing.\n\n## Build\n\nGuanaco is a CMake project that **patches and builds llama.cpp as a\nsubdirectory**. The patches add the `--guanaco-streaming` flag and the router\nhook; they are applied automatically at configure time, so a plain llama.cpp\ncheckout next to this repo is all you need.\n\n```sh\n# llama.cpp must be a sibling of this guanaco/ directory\nls ../llama.cpp # the upstream checkout to be patched in place\n\ncmake -B build -S .\ncmake --build build -j$(nproc)\n```\n\nThis produces a **patched** server and the streaming library:\n\n```\nbuild/bin/llama-server # patched; auto-enables Guanaco when requested\nbuild/bin/libguanaco.so\n```\n\n`io_uring` is used automatically when available (Linux 5.1+); otherwise it\nfalls back to a thread pool. Either way the behavior is identical.\n\n## Quick start (CPU / RAM-bound)\n\nThe usual case: you want a big MoE model on CPU with limited RAM.\n\n```sh\n./build/bin/llama-server \\\n -m model.gguf \\\n -ngl 0 `# CPU only; Guanaco owns the expert streaming` \\\n --guanaco-streaming `# enable disk streaming` \\\n --guanaco-max-experts 12 `# hot experts pinned per tensor (tune for RAM)`\n```\n\nOn a 48 GB box this runs a 120 GB MoE in well under 4 GB of RAM, with the\ncold experts page-faulting back from the GGUF as the router selects them.\n\nIt also coexists with GPU offload: e.g. run one model fully offloaded to a\nROCm/Vulkan device **and** a second model on CPU with `--guanaco-streaming`\nat the same time — they compete for different resources (VRAM vs. NVMe) and\neach runs at its native speed.\n\n## Enabling and tuning\n\n| Flag | Env override | Default | Meaning |\n|------|-------------|---------|---------|\n| `--guanaco-streaming` | `GUANACO_STREAMING=1` | off | Turn on expert disk streaming. |\n| `--guanaco-max-experts N` | `GUANACO_MAX_EXPERTS=N` | `-1` (auto → 8) | Hot experts pinned **per fused tensor**. Higher = more RAM, fewer disk reads; lower = less RAM, more streaming. |\n| (n/a) | `GUANACO_PILOT=0` | on | Cross-layer lookahead prefetch. Hints the next layer's likely experts while the current block computes. A wrong guess costs at most a wasted read. |\n| (n/a) | `GUANACO_PILOT_MASS=0.9` | `0.9` | Pilot reads only the top fraction of cumulative routing transition mass. Lower (e.g. `0.3`) trades a little hit-rate for far fewer pilot I/Os. `1.0` = full top-K. |\n| (n/a) | `GUANACO_IMATRIX=0` | on | Seed the hot set from a sibling `<model>.imatrix.gguf` before the first token. |\n| (n/a) | `GUANACO_IO_URING=0` | on (if available) | Force the thread-pool I/O backend instead of `io_uring`. |\n\n`GUANACO_MAX_EXPERTS` (env) overrides the `--guanaco-max-experts` flag, which\noverrides the auto default.\n\nDeep dive on the pinning knob and the resident/hit-rate tradeoff: see\n[`docs/PINNING.md`](docs/PINNING.md).\n\nMeasured results (including the 120 GB-in-4 GB run and the Guanaco-vs-GPU-offload\ncomparison): see [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md).\n\n## What you'll see (observability)\n\nGuanaco logs to stderr with the `[Guanaco HerdCache]` / `[Guanaco Storage]`\nprefix. The lines worth watching:\n\n```\n[Guanaco Storage] Model config: qwen35moe, experts=256, top_k=8, layers=40, hidden=2048\n[Guanaco Storage] Parsed 180 expert tensors across 41 layers (from 1 shard(s))\n[Guanaco HerdCache] pin budget=9 (dynamic per-tensor 6..9), experts pinned now=3372 ...\n```\n\n- **`dynamic per-tensor X..Y`** — the per-tensor pin budget is *not* uniform.\n Busy tensors (many distinct experts) get more slots; idle ones get fewer, so\n the same total RAM is spent where it is actually used.\n- **Periodic pin-cache line:**\n\n ```\n [Guanaco HerdCache] pin cache: 79% resident (52% hot-pinned),\n 1470657/1861566 expert accesses, distinct experts pinned=3372 across 120 tensors\n ```\n\n - `resident %` — share of router-selected experts that needed **no disk read**\n (pinned + already-warm). This is the headline number.\n - `hot-pinned %` — share served directly by a permanently pinned expert.\n - Low `resident %` and RAM to spare ⇒ raise `--guanaco-max-experts`.\n- **Final summary (on shutdown):**\n\n ```\n final: pin budget=9 (dynamic per-tensor 6..9), experts pinned now=3372 ...\n pin cache=79% resident (52% hot-pinned), 1863438 expert accesses over 74427 prefetch calls,\n evictions=416679, pilot: 36667 calls / 308618 slices,\n evict_hot=380364 evict_reread=1795 (0% churn), imatrix prior used\n ```\n\n - `evict_hot` / `evict_reread` and the **`churn %`** tell you how often an\n evicted expert was needed again soon. It should stay near **0%** — a high\n churn means the budget is too small for the model's routing spread.\n\n## Notes & caveats\n\n- **Requires an MoE architecture** (Mixtral, DeepSeek, Nemotron-H, Qwen-MoE,\n gpt-oss, LFM2, afmoe, …). Sparse-MoE layouts (MoE blocks at non-contiguous\n layers) are handled.\n- **Needs fast storage.** Cold experts are read from the GGUF on demand; an\n NVMe is strongly recommended. The whole point is that disk, not RAM, becomes\n the bottleneck — which is fine because the router lets us prefetch.\n- **Linux / `madvise` based.** `MADV_RANDOM` stops the OS from doing\n sequential read-ahead on expert regions; `MADV_DONTNEED` drops cold slices;\n `MADV_WILLNEED` faults hot ones in. Behavior is correct regardless: a dropped\n slice is re-read from the same file-backed address on next use.\n- **It is a patch on top of llama.cpp.** Build through this project's CMake (it\n patches `../llama.cpp` in place) rather than building llama.cpp directly, or\n the `--guanaco-streaming` flag and hook will be absent.\n- **Sharded GGUFs** (e.g. `-0000N-of-0000M.gguf`) are supported; the manifest\n is parsed per shard and the imatrix lookup tolerates the shard suffix.\n"}, "e6ca082bd7a60f927294a71567f1b2adb171941b": {"t": 1785771817.446692, "n": 6485, "h": "d4918930bc217f0b", "c": "# Guanaco (llama.cpp delta)\n\nGuanaco is a disk-streaming add-on for **llama.cpp MoE models**. It assumes you\nalready know how to build and run llama.cpp; this document only covers what is\n*different*.\n\nThe one idea: a Mixture-of-Experts model routes each token to a handful of\nexperts out of many, so most expert weights are idle at any moment. Plain\nllama.cpp memory-maps the whole GGUF and relies on the OS page cache to keep\nthe working set in RAM. Guanaco instead **keeps only a small set of \"hot\"\nexperts pinned in RAM and streams the rest from NVMe on demand**, so a model\nthat would never fit in system memory runs anyway — bounded by a knob you\ncontrol, not by the file size.\n\nDense (non-MoE) models are untouched: Guanaco detects no expert tensors and\ndoes nothing.\n\n## Build\n\nGuanaco is a CMake project that **patches and builds llama.cpp as a\nsubdirectory**. The patches add the `--guanaco-streaming` flag and the router\nhook; they are applied automatically at configure time, so a plain llama.cpp\ncheckout next to this repo is all you need.\n\n```sh\n# llama.cpp must be a sibling of this guanaco/ directory\nls ../llama.cpp # the upstream checkout to be patched in place\n\ncmake -B build -S .\ncmake --build build -j$(nproc)\n```\n\nThis produces a **patched** server and the streaming library:\n\n```\nbuild/bin/llama-server # patched; auto-enables Guanaco when requested\nbuild/bin/libguanaco.so\n```\n\n`io_uring` is used automatically when available (Linux 5.1+); otherwise it\nfalls back to a thread pool. Either way the behavior is identical.\n\n## Quick start (CPU / RAM-bound)\n\nThe usual case: you want a big MoE model on CPU with limited RAM.\n\n```sh\n./build/bin/llama-server \\\n -m model.gguf \\\n -ngl 0 `# CPU only; Guanaco owns the expert streaming` \\\n --guanaco-streaming `# enable disk streaming` \\\n --guanaco-max-experts 12 `# hot experts pinned per tensor (tune for RAM)`\n```\n\nOn a 48 GB box this runs a 120 GB MoE in well under 4 GB of RAM, with the\ncold experts page-faulting back from the GGUF as the router selects them.\n\nIt also coexists with GPU offload: e.g. run one model fully offloaded to a\nROCm/Vulkan device **and** a second model on CPU with `--guanaco-streaming`\nat the same time — they compete for different resources (VRAM vs. NVMe) and\neach runs at its native speed.\n\n## Enabling and tuning\n\n| Flag | Env override | Default | Meaning |\n|------|-------------|---------|---------|\n| `--guanaco-streaming` | `GUANACO_STREAMING=1` | off | Turn on expert disk streaming. |\n| `--guanaco-max-experts N` | `GUANACO_MAX_EXPERTS=N` | `-1` (auto → 8) | Hot experts pinned **per fused tensor**. Higher = more RAM, fewer disk reads; lower = less RAM, more streaming. |\n| (n/a) | `GUANACO_PILOT=0` | on | Cross-layer lookahead prefetch. Hints the next layer's likely experts while the current block computes. A wrong guess costs at most a wasted read. |\n| (n/a) | `GUANACO_PILOT_MASS=0.9` | `0.9` | Pilot reads only the top fraction of cumulative routing transition mass. Lower (e.g. `0.3`) trades a little hit-rate for far fewer pilot I/Os. `1.0` = full top-K. |\n| (n/a) | `GUANACO_IMATRIX=0` | on | Seed the hot set from a sibling `<model>.imatrix.gguf` before the first token. |\n| (n/a) | `GUANACO_IO_URING=0` | on (if available) | Force the thread-pool I/O backend instead of `io_uring`. |\n\n`GUANACO_MAX_EXPERTS` (env) overrides the `--guanaco-max-experts` flag, which\noverrides the auto default.\n\nDeep dive on the pinning knob and the resident/hit-rate tradeoff: see\n[`docs/PINNING.md`](docs/PINNING.md).\n\nMeasured results (including the 120 GB-in-4 GB run and the Guanaco-vs-GPU-offload\ncomparison): see [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md).\n\n## What you'll see (observability)\n\nGuanaco logs to stderr with the `[Guanaco HerdCache]` / `[Guanaco Storage]`\nprefix. The lines worth watching:\n\n```\n[Guanaco Storage] Model config: qwen35moe, experts=256, top_k=8, layers=40, hidden=2048\n[Guanaco Storage] Parsed 180 expert tensors across 41 layers (from 1 shard(s))\n[Guanaco HerdCache] pin budget=9 (dynamic per-tensor 6..9), experts pinned now=3372 ...\n```\n\n- **`dynamic per-tensor X..Y`** — the per-tensor pin budget is *not* uniform.\n Busy tensors (many distinct experts) get more slots; idle ones get fewer, so\n the same total RAM is spent where it is actually used.\n- **Periodic pin-cache line:**\n\n ```\n [Guanaco HerdCache] pin cache: 79% resident (52% hot-pinned),\n 1470657/1861566 expert accesses, distinct experts pinned=3372 across 120 tensors\n ```\n\n - `resident %` — share of router-selected experts that needed **no disk read**\n (pinned + already-warm). This is the headline number.\n - `hot-pinned %` — share served directly by a permanently pinned expert.\n - Low `resident %` and RAM to spare ⇒ raise `--guanaco-max-experts`.\n- **Final summary (on shutdown):**\n\n ```\n final: pin budget=9 (dynamic per-tensor 6..9), experts pinned now=3372 ...\n pin cache=79% resident (52% hot-pinned), 1863438 expert accesses over 74427 prefetch calls,\n evictions=416679, pilot: 36667 calls / 308618 slices,\n evict_hot=380364 evict_reread=1795 (0% churn), imatrix prior used\n ```\n\n - `evict_hot` / `evict_reread` and the **`churn %`** tell you how often an\n evicted expert was needed again soon. It should stay near **0%** — a high\n churn means the budget is too small for the model's routing spread.\n\n## Notes & caveats\n\n- **Requires an MoE architecture** (Mixtral, DeepSeek, Nemotron-H, Qwen-MoE,\n gpt-oss, LFM2, afmoe, …). Sparse-MoE layouts (MoE blocks at non-contiguous\n layers) are handled.\n- **Needs fast storage.** Cold experts are read from the GGUF on demand; an\n NVMe is strongly recommended. The whole point is that disk, not RAM, becomes\n the bottleneck — which is fine because the router lets us prefetch.\n- **Linux / `madvise` based.** `MADV_RANDOM` stops the OS from doing\n sequential read-ahead on expert regions; `MADV_DONTNEED` drops cold slices;\n `MADV_WILLNEED` faults hot ones in. Behavior is correct regardless: a dropped\n slice is re-read from the same file-backed address on next use.\n- **It is a patch on top of llama.cpp.** Build through this project's CMake (it\n patches `../llama.cpp` in place) rather than building llama.cpp directly, or\n the `--guanaco-streaming` flag and hook will be absent.\n- **Sharded GGUFs** (e.g. `-0000N-of-0000M.gguf`) are supported; the manifest\n is parsed per shard and the imatrix lookup tolerates the shard suffix.\n"}, "0eee2ae8d972c7007364bcb6d074aaf6d61b80b4": {"t": 1785771818.6943805, "n": 6341, "h": "ccadd8de66ca6203", "c": "# Guanaco (llama.cpp delta)\n\nGuanaco is a disk-streaming add-on for **llama.cpp MoE models**. It assumes you\nalready know how to build and run llama.cpp; this document only covers what is\n*different*.\n\nThe one idea: a Mixture-of-Experts model routes each token to a handful of\nexperts out of many, so most expert weights are idle at any moment. Plain\nllama.cpp memory-maps the whole GGUF and relies on the OS page cache to keep\nthe working set in RAM. Guanaco instead **keeps only a small set of \"hot\"\nexperts pinned in RAM and streams the rest from NVMe on demand**, so a model\nthat would never fit in system memory runs anyway — bounded by a knob you\ncontrol, not by the file size.\n\nDense (non-MoE) models are untouched: Guanaco detects no expert tensors and\ndoes nothing.\n\n## Build\n\nGuanaco is a CMake project that **patches and builds llama.cpp as a\nsubdirectory**. The patches add the `--guanaco-streaming` flag and the router\nhook; they are applied automatically at configure time, so a plain llama.cpp\ncheckout next to this repo is all you need.\n\n```sh\n# llama.cpp must be a sibling of this guanaco/ directory\nls ../llama.cpp # the upstream checkout to be patched in place\n\ncmake -B build -S .\ncmake --build build -j$(nproc)\n```\n\nThis produces a **patched** server and the streaming library:\n\n```\nbuild/bin/llama-server # patched; auto-enables Guanaco when requested\nbuild/bin/libguanaco.so\n```\n\n`io_uring` is used automatically when available (Linux 5.1+); otherwise it\nfalls back to a thread pool. Either way the behavior is identical.\n\n## Quick start (CPU / RAM-bound)\n\nThe usual case: you want a big MoE model on CPU with limited RAM.\n\n```sh\n./build/bin/llama-server \\\n -m model.gguf \\\n -ngl 0 `# CPU only; Guanaco owns the expert streaming` \\\n --guanaco-streaming `# enable disk streaming` \\\n --guanaco-max-experts 12 `# hot experts pinned per tensor (tune for RAM)`\n```\n\nOn a 48 GB box this runs a 120 GB MoE in well under 4 GB of RAM, with the\ncold experts page-faulting back from the GGUF as the router selects them.\n\nIt also coexists with GPU offload: e.g. run one model fully offloaded to a\nROCm/Vulkan device **and** a second model on CPU with `--guanaco-streaming`\nat the same time — they compete for different resources (VRAM vs. NVMe) and\neach runs at its native speed.\n\n## Enabling and tuning\n\n| Flag | Env override | Default | Meaning |\n|------|-------------|---------|---------|\n| `--guanaco-streaming` | `GUANACO_STREAMING=1` | off | Turn on expert disk streaming. |\n| `--guanaco-max-experts N` | `GUANACO_MAX_EXPERTS=N` | `-1` (auto → 8) | Hot experts pinned **per fused tensor**. Higher = more RAM, fewer disk reads; lower = less RAM, more streaming. |\n| (n/a) | `GUANACO_PILOT=0` | on | Cross-layer lookahead prefetch. Hints the next layer's likely experts while the current block computes. A wrong guess costs at most a wasted read. |\n| (n/a) | `GUANACO_PILOT_MASS=0.9` | `0.9` | Pilot reads only the top fraction of cumulative routing transition mass. Lower (e.g. `0.3`) trades a little hit-rate for far fewer pilot I/Os. `1.0` = full top-K. |\n| (n/a) | `GUANACO_IMATRIX=0` | on | Seed the hot set from a sibling `<model>.imatrix.gguf` before the first token. |\n| (n/a) | `GUANACO_IO_URING=0` | on (if available) | Force the thread-pool I/O backend instead of `io_uring`. |\n\n`GUANACO_MAX_EXPERTS` (env) overrides the `--guanaco-max-experts` flag, which\noverrides the auto default.\n\nDeep dive on the pinning knob and the resident/hit-rate tradeoff: see\n[`docs/PINNING.md`](docs/PINNING.md).\n\n## What you'll see (observability)\n\nGuanaco logs to stderr with the `[Guanaco HerdCache]` / `[Guanaco Storage]`\nprefix. The lines worth watching:\n\n```\n[Guanaco Storage] Model config: qwen35moe, experts=256, top_k=8, layers=40, hidden=2048\n[Guanaco Storage] Parsed 180 expert tensors across 41 layers (from 1 shard(s))\n[Guanaco HerdCache] pin budget=9 (dynamic per-tensor 6..9), experts pinned now=3372 ...\n```\n\n- **`dynamic per-tensor X..Y`** — the per-tensor pin budget is *not* uniform.\n Busy tensors (many distinct experts) get more slots; idle ones get fewer, so\n the same total RAM is spent where it is actually used.\n- **Periodic pin-cache line:**\n\n ```\n [Guanaco HerdCache] pin cache: 79% resident (52% hot-pinned),\n 1470657/1861566 expert accesses, distinct experts pinned=3372 across 120 tensors\n ```\n\n - `resident %` — share of router-selected experts that needed **no disk read**\n (pinned + already-warm). This is the headline number.\n - `hot-pinned %` — share served directly by a permanently pinned expert.\n - Low `resident %` and RAM to spare ⇒ raise `--guanaco-max-experts`.\n- **Final summary (on shutdown):**\n\n ```\n final: pin budget=9 (dynamic per-tensor 6..9), experts pinned now=3372 ...\n pin cache=79% resident (52% hot-pinned), 1863438 expert accesses over 74427 prefetch calls,\n evictions=416679, pilot: 36667 calls / 308618 slices,\n evict_hot=380364 evict_reread=1795 (0% churn), imatrix prior used\n ```\n\n - `evict_hot` / `evict_reread` and the **`churn %`** tell you how often an\n evicted expert was needed again soon. It should stay near **0%** — a high\n churn means the budget is too small for the model's routing spread.\n\n## Notes & caveats\n\n- **Requires an MoE architecture** (Mixtral, DeepSeek, Nemotron-H, Qwen-MoE,\n gpt-oss, LFM2, afmoe, …). Sparse-MoE layouts (MoE blocks at non-contiguous\n layers) are handled.\n- **Needs fast storage.** Cold experts are read from the GGUF on demand; an\n NVMe is strongly recommended. The whole point is that disk, not RAM, becomes\n the bottleneck — which is fine because the router lets us prefetch.\n- **Linux / `madvise` based.** `MADV_RANDOM` stops the OS from doing\n sequential read-ahead on expert regions; `MADV_DONTNEED` drops cold slices;\n `MADV_WILLNEED` faults hot ones in. Behavior is correct regardless: a dropped\n slice is re-read from the same file-backed address on next use.\n- **It is a patch on top of llama.cpp.** Build through this project's CMake (it\n patches `../llama.cpp` in place) rather than building llama.cpp directly, or\n the `--guanaco-streaming` flag and hook will be absent.\n- **Sharded GGUFs** (e.g. `-0000N-of-0000M.gguf`) are supported; the manifest\n is parsed per shard and the imatrix lookup tolerates the shard suffix.\n"}}}} |