OpenCode
68a9071cf1
Local branch had diverged from origin/master (sibling commits on the same base). Rewrote local history linearly on top of origin/master, folding in all local content: exa/hot-platform discovery hunters, leaks ledger, nightly sweep orchestrator, updated .gitignore and skill docs, plus the latest hunt output and vault state. Remote-only files (vault keys, channel scripts) were restored rather than dropped, so the resulting tree is a full union of both sides.
1 line
39 KiB
JSON
1 line
39 KiB
JSON
{"repo": "caestrada1103/ai-cluster", "files": {"README.md": {"69916006810d500dcb31c31a84d660cd893f05af": {"t": 1786068577.4125512, "n": 36082, "h": "e578327f9bbc9429", "c": "# AI Cluster - Distributed Multi-GPU Inference System\n\n[](https://www.rust-lang.org)\n[](https://www.python.org)\n[](https://rocm.docs.amd.com)\n[](https://developer.nvidia.com/cuda-toolkit)\n[](LICENSE)\n[](docs/architecture.md)\n\n## Run it\n\nOne command, then open a browser:\n\n```bash\n./scripts/run-local.sh --ui\n```\n\nChat UI at **http://localhost:8080**, API at **http://localhost:8000**. Ctrl-C\nstops everything. Pick a model in the UI's model dropdown; the coordinator\nloads it on first use.\n\nWithout `--ui` it starts the API only. `--build` rebuilds the worker first,\n`--model NAME` preloads a model, `--help` lists everything. First-time setup\nis in [Installation](#installation); `--ui` also needs Docker.\n\n## 📋 Table of Contents\n\n- [Overview](#overview)\n- [Features](#features)\n- [Architecture](#architecture)\n- [Security: Secure by Default](#security-secure-by-default)\n- [Quick Start](#quick-start)\n- [Documentation](#documentation)\n- [Supported Models](#supported-models)\n- [Performance](#performance)\n- [Use Cases](#use-cases)\n- [Use on Real Scenarios](#use-on-real-scenarios)\n- [Installation](#installation)\n- [Configuration](#configuration)\n- [API Reference](#api-reference)\n- [Monitoring](#monitoring)\n- [Contributing](#contributing)\n- [License](#license)\n- [Acknowledgments](#acknowledgments)\n\n---\n\n## Overview\n\n**AI Cluster** is a production-ready distributed system for running large language models (LLMs) across multiple GPUs and machines. It provides a unified API for model inference while automatically handling the complexities of distribution, parallelism, and resource management.\n\n[**Why use AI Cluster? Read our Non-Technical Introduction.**](docs/problem_solution.md)\n\nWhether you have a single workstation with multiple GPUs or a rack of servers, AI Cluster scales to meet your needs while maintaining low latency and high throughput.\n\n### Why AI Cluster?\n\n- **🚀 High Performance**: Optimized for both AMD and NVIDIA GPUs with custom kernels\n- **🔧 Hardware Agnostic**: Same code runs on AMD, NVIDIA, or CPU\n- **📈 Elastic Scaling**: Add or remove workers without downtime\n- **🛡️ Production Ready**: Built-in monitoring, fault tolerance, and security\n- **🎯 Easy to Use**: Simple REST API compatible with OpenAI's interface\n- **💪 Powerful Parallelism**: Pipeline, tensor, data, and expert parallelism\n\n---\n\n## Features\n\n### Core Features\n\n| Feature | Description |\n|---------|-------------|\n| **Multi-GPU Support** | Run models across multiple AMD (ROCm) or NVIDIA (CUDA) GPUs via multiple workers |\n| **Dynamic Model Loading** | Load/unload models at runtime without restart |\n| **REST API** | OpenAI-compatible API for easy integration |\n| **Streaming** | Stream tokens as they're generated |\n| **Quantization** | Planned — FP16/INT8/INT4/FP8 (weights load as FP32 today; non-NONE requests are rejected) |\n\n### Advanced Features\n\n| Feature | Description |\n|---------|-------------|\n| **Circuit Breakers** | Prevents cascading failures |\n| **Request Queuing** | Priority-based request handling |\n| **Affinity Routing** | Session persistence for chatbots |\n| **Prometheus Metrics** | Comprehensive monitoring |\n| **Grafana Dashboards** | Pre-built visualizations |\n| **Kubernetes Support** | Planned — no manifests ship yet |\n\n### Roadmap & Planned Optimizations\n\n**Status legend:**\n\n| Badge | Meaning |\n|-------|---------|\n| ✅ Done (Completed) | Fully implemented and wired end-to-end |\n| 🔶 Done (Partially) | Core logic implemented; integration or secondary features missing |\n| 🔄 In-Progress | Actively being developed on the current branch |\n| 🔲 To-Do | Not yet started |\n\n| Feature | Status | Notes |\n|---------|--------|-------|\n| **KV Cache** | ✅ Done (Completed) | Per-layer `KvEntry<B>` / `KvCache<B>`; prefill + autoregressive decode fully wired in Llama; TP-sharded `TpKvCache<B>` also implemented |\n| **Tensor Parallelism** | 🔶 Done (Partially) | `tensor_parallel_llama_prefill` and `tensor_parallel_llama_decode_step` implemented with Megatron-LM style column/row split and `AllReduce` abstraction; not yet wired to the gRPC service layer |\n| **Pipeline Parallelism** | 🔶 Done (Partially) | `pipeline_parallel_llama_forward` implements layer-chunk partitioning; not yet wired to the gRPC service layer |\n| **Expert Parallelism (MoE)** | 🔄 In-Progress | `DeepSeekMoE` sparse top-k routing implemented in `worker/models/deepseek.rs`; distributed expert routing across GPUs is a stub returning an error |\n| **Data Parallelism** | ✅ Done (Completed) | Fully operational — deploy multiple independent workers (one per GPU) behind the coordinator's load balancer |\n| **Cross-Node ggml-RPC Split (GGUF)** | 🔶 Done (Partially) | Worker supervises a real `ggml-rpc-server` peer and threads `--rpc`/`--tensor-split` into the lead's `llama-server` (opt-in `llamacpp-rpc` cargo feature). Coordinator resolves `[models.X.distributed]`, loads peers before the lead, unloads lead-first, and rolls back a failed load. Both sides are unit-tested but the integrated path has not been exercised on real hardware. See `docs/configuration.md`. |\n| **Continuous Batching** | 🔲 To-Do | High-throughput inference with dynamic batching |\n| **Paged KV Cache** | 🔲 To-Do | vLLM-style memory-efficient paged attention; basic KV cache is done, the paged variant is not |\n| **Speculative Decoding** | 🔲 To-Do | 2-3x generation speedup with draft model |\n| **Flash Attention** | 🔲 To-Do | Fast and memory-efficient exact attention |\n| **Distributed Tracing** | 🔲 To-Do | Jaeger integration for end-to-end request tracing |\n| **Extended Infrastructure** | 🔲 To-Do | Redis (caching), MinIO (model storage), Vault (secrets), Elastic (logs) |\n\n---\n\n---\n\n## Tech Stack\n\n- **Coordinator**: Python (FastAPI, gRPC)\n- **Worker**: Rust (Burn Framework + llama.cpp, Tokio, Tonic)\n- **Inference Engines**: **llama.cpp/GGUF** (`llama-cpp-2 =0.1.150`) — **recommended engine** for quantized inference on consumer NVIDIA/AMD GPUs (native Q4_K_M/Q5_K_M/Q8_0 quantization; opt-in via the `llamacpp` cargo feature) + **Burn 0.19** (safetensors, FP32, the **default** cargo build feature) — experimental/reference engine, no quantization yet\n- **Communication**: gRPC (Inter-service), REST API (Client-facing)\n- **GPU Acceleration**:\n - **AMD**: ROCm (HIP)\n - **NVIDIA**: CUDA\n- **Infrastructure**: Docker, Prometheus, Grafana\n- **Planned Infrastructure**: Jaeger, Redis, MinIO, Vault, Elastic\n\n## Architecture\n\n```\n┌─────────────────────────────────────────────────────────────────────┐\n│ Client Applications │\n│ (REST API, Web UI, CLI, SDKs) │\n└────────────────────────────────┬────────────────────────────────────┘\n │\n ▼\n┌─────────────────────────────────────────────────────────────────────┐\n│ Coordinator (single instance) │\n│ FastAPI REST · worker discovery · routing · registry │\n│ (HA replicas + leader election are on the roadmap, not built) │\n└────────────────────────────────┬────────────────────────────────────┘\n │\n ┌────────────┴────────────┐\n │ │\n ▼ ▼\n┌───────────────────────────────────┐ ┌───────────────────────────────────┐\n│ Worker Pool (AMD) │ │ Worker Pool (NVIDIA) │\n│ ┌─────────────┐ ┌─────────────┐ │ │ ┌─────────────┐ ┌─────────────┐ │\n│ │ Worker AMD │ │ Worker AMD │ │ │ │Worker NVIDIA│ │Worker NVIDIA│ │\n│ │ GPU 0-3 │ │ GPU 4-7 │ │ │ │ GPU 0-3 │ │ GPU 4-7 │ │\n│ └─────────────┘ └─────────────┘ │ │ └─────────────┘ └─────────────┘ │\n│ │ │ │\n│ ┌─────────────┐ ┌─────────────┐ │ │ ┌─────────────┐ ┌─────────────┐ │\n│ │ Worker AMD │ │ Worker AMD │ │ │ │Worker NVIDIA│ │Worker NVIDIA│ │\n│ │ CPU Only │ │ Mixture │ │ │ │ CPU Only │ │ Mixture │ │\n│ └─────────────┘ └─────────────┘ │ │ └─────────────┘ └─────────────┘ │\n└───────────────────────────────────┘ └───────────────────────────────────┘\n │ │\n └────────────┬────────────┘\n │\n ▼\n┌─────────────────────────────────────────────────────────────────────┐\n│ Infrastructure Layer (shipped today) │\n│ ┌─────────────┐ ┌─────────────┐ │\n│ │ Prometheus │ │ Grafana │ │\n│ └─────────────┘ └─────────────┘ │\n│ Planned: Redis · MinIO · Jaeger · Consul · Vault · Elastic │\n└─────────────────────────────────────────────────────────────────────┘\n```\n\nFor detailed architecture information, see the [Architecture Guide](docs/architecture.md).\n\n---\n\n## Security: Secure by Default\n\nEverything binds to loopback until you explicitly open it up:\n\n- Worker gRPC and spawned `llama-server` children bind `127.0.0.1` by\n default, on bare metal and in containers alike.\n- `WORKER_GRPC_AUTH_TOKEN` is a shared secret required on every gRPC call\n once set; the worker logs a warning if it ends up bound to a non-loopback\n address with no token configured.\n- The coordinator refuses to start bound to a non-loopback address unless\n `COORDINATOR_API_KEYS` is set.\n- Individual keys can carry a role (`admin`/`user`) and a model scope via\n `COORDINATOR_API_KEY_FILE`, layered on top of `COORDINATOR_API_KEYS`. See\n [Configuration Guide](docs/configuration.md) for the TOML shape and the\n compatibility guarantee for deployments that don't set it.\n- `docker compose up` publishes only the coordinator (8000), Grafana (3000),\n and Prometheus (9099) — worker gRPC/metrics, `llama-server`, and\n Open-WebUI ports stay off the host. Opt in with\n `docker compose -f docker-compose.yml -f docker-compose.expose-ports.yml up`.\n\nContainers open the worker binds up via `WORKER_GRPC_BIND_HOST` /\n`LLAMASERVER_BIND_HOST` (both default to `0.0.0.0` in `docker-compose.yml`);\n`config/worker.toml` itself never widens a bind. See\n[Deployment Guide](docs/deployment.md) and\n[Configuration Guide](docs/configuration.md) for the full picture.\n\n---\n\n## Quick Start\n\n> **Recommended for consumer GPUs**: the steps below bring up the full stack\n> using the default Burn/wgpu build (FP32, single GPU per worker) so you can\n> get everything running quickly. For **quantized** models on consumer\n> NVIDIA/AMD GPUs (~8-16 GB VRAM) — the project's primary goal — build the\n> worker with the **llama.cpp/GGUF engine** instead (`--features llamacpp`\n> [+ `llamacpp-vulkan`/`llamacpp-cuda`]) and load a `[models.X.gguf]` entry;\n> see [Adding a GGUF model](#adding-a-gguf-model-llamacpp-engine-recommended)\n> below for the recommended end-to-end example.\n\n### Method 1: Docker Compose\n\nThe easiest way to get started is using Docker Compose, which handles all dependencies automatically.\n\n```bash\n# 1. Clone the repository\ngit clone https://github.com/caestrada1103/ai-cluster.git\ncd ai-cluster\n\n# 2. Configure Environment. REQUIRED:\n# - GRAFANA_ADMIN_PASSWORD (compose refuses to start Grafana without it)\n# - COORDINATOR_API_KEYS (the coordinator's port is published to the host,\n# a non-loopback bind, so it refuses to start with no keys set — secure\n# by default; generate one with `openssl rand -hex 32`)\n# - COMPOSE_PROFILES matching your GPU vendor (no profile = no worker starts)\n# HF_TOKEN is needed for gated models like Llama 3.\ncp .env.example .env\n# Edit .env: set GRAFANA_ADMIN_PASSWORD, COORDINATOR_API_KEYS, COMPOSE_PROFILES, HF_TOKEN\n\n# 3. Build and start with Docker Compose\ndocker compose up -d --build\n\n# 4. Check that everything is running (/health needs no key)\ncurl http://localhost:8000/health\n\n# 5. Run your first inference (Model will auto-download and load)\nH='Authorization: Bearer <your COORDINATOR_API_KEYS value>'\ncurl -H \"$H\" -X POST http://localhost:8000/v1/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"deepseek-7b\",\n \"prompt\": \"Hello, how are you?\",\n \"max_tokens\": 50\n }'\n```\n\nSee [GPU variants](docs/deployment.md#gpu-variants) for the exact\n`COMPOSE_PROFILES` value per vendor.\n\n### Method 2: Local Execution (Native GPU) — verified\n\n**Shortcut:** once the one-time setup below is done, `./scripts/run-local.sh`\nstarts the worker and coordinator together, waits for both to be ready, prints\nworking `curl` examples, and stops both on Ctrl-C. `--build` rebuilds the\nworker first; `--model NAME` loads a model once healthy. The manual steps\nfollow for when you want the two processes in separate terminals.\n\nRuns entirely on loopback (secure by default) — no ports leave the machine.\nThis example uses `qwen3.6-35b-a3b-gguf`, a `llamaserver`-engine model, so it\nalso needs a `llama-server` binary on `PATH` (build steps in\n[Deployment → llama-server for agentic serving](docs/deployment.md); Node\n20+ is required for that build only, not for anything below).\n\n```bash\n# one-time\nsudo apt-get install -y protobuf-compiler libclang-dev libssl-dev\ncurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --no-modify-path\npython3 -m venv .venv && . .venv/bin/activate && pip install -r coordinator/requirements-dev.txt\ncd worker && cargo build --release --features wgpu,llamacpp,llamacpp-cuda && cd ..\n\n# terminal 1\nexport PATH=\"$HOME/.local/bin:$PATH\" # so the worker finds llama-server\nRUST_LOG=info ./target/release/ai-worker --port 50051\n\n# terminal 2\n. .venv/bin/activate\nexport COORDINATOR_STATIC_WORKERS='[\"localhost:50051\"]'\nexport COORDINATOR_API_KEYS='sk-local-dev'\nuvicorn coordinator.main:app --host 127.0.0.1 --port 8000\n\n# use it\nH='Authorization: Bearer sk-local-dev'\ncurl -H \"$H\" -X POST localhost:8000/v1/models/load -H 'Content-Type: application/json' \\\n -d '{\"model_name\":\"qwen3.6-35b-a3b-gguf\"}'\ncurl -H \"$H\" -X POST localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \\\n -d '{\"model\":\"qwen3.6-35b-a3b-gguf\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}],\"max_tokens\":200}'\n```\n\nThe worker binary lands at the **workspace root** `target/release/ai-worker`\n— not `worker/target/` — because `worker` is a Cargo workspace member. Swap\n`llamacpp-cuda` for `llamacpp-vulkan` on AMD/Intel, or drop it for CPU-only\nGGUF inference; add `cuda`/`rocm` in place of `wgpu` for a native (non-Vulkan)\nBurn backend. See [Adding a GGUF model](#adding-a-gguf-model-llamacpp-engine-recommended)\nfor a lighter `llamacpp`-engine example instead of `llamaserver`.\n\n---\n\n## Documentation\n\n| Document | Description |\n|----------|-------------|\n| [Architecture Guide](docs/architecture.md) | System design, components, data flow |\n| [API Reference](docs/api_reference.md) | Complete API documentation with examples |\n| [Configuration Guide](docs/configuration.md) | All configuration options explained |\n| [Deployment Guide](docs/deployment.md) | Single machine, cluster, Kubernetes, cloud |\n| [Troubleshooting](docs/troubleshooting.md) | Common issues and solutions |\n\n### Quick Links\n\n- [Installation Guide](#installation)\n- [Configuration Examples](#configuration)\n- [API Examples](#api-reference)\n- [Sizing Examples (measured on real hardware)](docs/configuration.md#sizing-examples-measured-on-real-hardware)\n- [Security: Secure by Default](#security-secure-by-default)\n\n---\n\n## Supported Models\n\nAI Cluster runs models through two inference engines: **llama.cpp/GGUF**\n(recommended — quantized, runs on consumer NVIDIA + AMD GPUs) and\n**Burn/safetensors** (experimental FP32 reference engine, single GPU per\nworker).\n\n### GGUF models via llama.cpp (recommended)\n\nThis is the recommended way to run models on consumer-grade GPUs (~8-16 GB\nVRAM): weights are loaded **quantized** (Q4_K_M, Q5_K_M, Q8_0, …) so larger\nmodels fit in limited VRAM, and inference runs on both NVIDIA (CUDA/Vulkan)\nand AMD (ROCm/Vulkan). It's opt-in — build the worker with `--features\nllamacpp` (see [Adding a GGUF model](#adding-a-gguf-model-llamacpp-engine-recommended)\nbelow). Splitting one GGUF model across several consumer GPUs is on the\nworker roadmap; today one GGUF model loads per worker process.\n\n| Model | Status | Notes |\n|-------|--------|-------|\n| **Any GGUF checkpoint** (e.g. Qwen2.5, Llama 3.1, Qwen2.5-Coder, …) | ✅ Implemented (opt-in: build worker with `--features llamacpp` [+ `llamacpp-vulkan`/`llamacpp-cuda`]) | Native quantization: Q4_K_M, Q5_K_M, Q8_0, …; NVIDIA + AMD; multi-GPU split upcoming |\n\n### Agentic serving via llama-server (tool calling)\n\nA third serving mode, `engine = \"llamaserver\"`, runs OpenAI/Anthropic **tool\ncalling** for coding agents (Claude Code, Cline, aider, …): the worker supervises\na `llama-server` process per model and the coordinator proxies agentic HTTP\nrequests (including `/v1/messages`) straight to it. The default Docker worker\nimage bundles the `llama-server` binary; bare-metal setup, the pinned llama.cpp\nversion, and the coordinator↔worker port requirements are covered in the\n[Deployment Guide](docs/deployment.md). The number of concurrent\nconversation slots is `[models.X.llamaserver] instances` in\n`config/models.toml` (default 1), also overridable per request with\n`POST /v1/models/load {\"instances\": N}` — see\n[Configuration Guide](docs/configuration.md). See\n[Use on Real Scenarios](#use-on-real-scenarios) for the end-to-end Claude Code\nwalkthrough.\n\n### Burn / safetensors models (experimental FP32 reference)\n\nThe Burn engine is the **default** cargo build feature (`--features\nwgpu|cuda|rocm`) and serves as the experimental / reference path: it loads\nfull-precision **FP32** safetensors weights — it does not quantize (non-`none`\nquantization requests are rejected) — and runs a model on a **single GPU** per\nworker. Tensor/pipeline parallelism exist in `worker/src/parallelism.rs` but\nare not yet wired to inference.\n\n| Model Family | Sizes | Status | Notes |\n|--------------|-------|--------|-------|\n| **Llama 3** | 8B, 70B | ✅ Implemented | reference architecture |\n| **Qwen 2.5 Coder** | 32B | ✅ Implemented | GQA + RoPE + SwiGLU (Qwen3 not yet supported) |\n| **DeepSeek (dense)** | 7B, 67B | ✅ Implemented | loads via the Llama-layout path |\n| **DeepSeek (MoE/V3)** | 671B | 🔶 Model code present | V3 routing (sigmoid/MLA) not implemented |\n| **Mistral** | 7B | 🔶 Model code present, not wired to loader | no TextGeneration/tokenizer/KV cache yet |\n| **Mixtral** | 8x7B | 🔲 Planned | |\n| **Gemma** | 2B, 7B | 🔲 Planned | |\n| **Phi** | 2, 3-mini | 🔲 Planned | |\n\n### Adding a GGUF model (llama.cpp engine, recommended)\n\n```toml\n# config/models.toml — no architecture block needed (read from GGUF metadata)\n[models.\"qwen2.5-0.5b-gguf\"]\nfamily = \"qwen\"\nparameters = \"0.5B\"\nmin_memory_gb = 1\nengine = \"llamacpp\"\n\n[models.\"qwen2.5-0.5b-gguf\".gguf]\nrepo_id = \"Qwen/Qwen2.5-0.5B-Instruct-GGUF\"\nfile = \"qwen2.5-0.5b-instruct-q4_k_m.gguf\"\nn_gpu_layers = -1 # -1 = all layers on GPU\nn_ctx = 4096\n```\n\nThe worker must be built with the engine enabled: `cargo build --release --features wgpu,llamacpp` (add `llamacpp-vulkan` or `llamacpp-cuda` for GPU offload), or `docker build -f docker/Dockerfile.worker --build-arg WORKER_FEATURES=\"llamacpp,llamacpp-vulkan\" .`. Then load and run it like any other model:\n\n```bash\ncurl -X POST http://localhost:8000/v1/models/load \\\n -d '{\"model_name\": \"qwen2.5-0.5b-gguf\"}'\n\ncurl -X POST http://localhost:8000/v1/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"qwen2.5-0.5b-gguf\",\n \"prompt\": \"Hello, how are you?\",\n \"max_tokens\": 50\n }'\n```\n\n### Adding a Burn/safetensors model (reference engine)\n\n```python\n# 1. Convert your model\npython scripts/convert_model.py your-username/your-model --output ./models/\n\n# 2. Add to model registry (config/models.toml)\n[models.\"your-model\"]\nfamily = \"custom\"\nparameters = \"7B\"\nmin_memory_gb = 16\n\n[models.\"your-model\".architecture]\nnum_layers = 32\nhidden_size = 4096\n# ... model architecture\n\n# 3. Load and use\ncurl -X POST http://localhost:8000/v1/models/load \\\n -d '{\"model_name\": \"your-model\"}'\n```\n\n---\n\n## Performance\n\n> **Disclaimer**: The benchmarks and metrics below represent **projected goals** leveraging our planned future optimizations (such as Continuous Batching, Paged KV Cache, and advanced Tensor Parallelism). Current single-worker baseline performance will vary depending on your hardware environment.\n\n### Projected Benchmarks\n\n| Model | GPUs | Batch Size | Projected Tokens/sec | Projected Latency (P95) |\n|-------|------|------------|----------------------|-------------------------|\n| DeepSeek-7B | 1x AMD 9060 XT | 1 | 45 | 120ms |\n| DeepSeek-7B | 1x AMD 9060 XT | 8 | 210 | 380ms |\n| DeepSeek-7B | 2x AMD 9060 XT | 16 | 410 | 420ms |\n| DeepSeek-67B | 4x AMD 9060 XT | 1 | 12 | 450ms |\n| Llama-3-8B | 1x NVIDIA T4 | 1 | 52 | 105ms |\n| Llama-3-8B | 1x NVIDIA T4 | 8 | 245 | 350ms |\n| Llama-3-70B | 4x NVIDIA A100 | 1 | 28 | 210ms |\n| Mixtral-8x7B | 2x NVIDIA A100 | 1 | 18 | 320ms |\n\n### Scaling Efficiency (Data Parallel Multi-Worker)\n\n```\nThroughput vs. Number of GPUs (DeepSeek-7B)\n─────────────────────────────────────────────\n4 GPUs ──────────────────▒ 410 tok/s (91%)\n3 GPUs ────────────────▒ 320 tok/s (94%)\n2 GPUs ─────────────▒ 210 tok/s (97%)\n1 GPU ───────▒ 100 tok/s (100%)\n 0 100 200 300 400 500\n Tokens per second\n```\n\n### Roadmap Optimizations Impact\n\nOnce fully implemented, these optimizations are projected to provide the following benefits for constrained hardware settings:\n- **Continuous Batching**: 2-3x throughput improvement\n- **Paged KV Cache**: 50-70% memory reduction\n- **Flash Attention**: 2-4x faster attention\n- **Speculative Decoding**: 2-3x faster generation\n\n**Quantization** isn't on this roadmap list — it's available **today**, but only via the **llama.cpp/GGUF engine** (Q4_K_M, Q5_K_M, Q8_0, …; opt-in `--features llamacpp`), which typically cuts memory ~50-75% vs. full precision depending on the quant level. The default Burn/safetensors path still loads weights as FP32 and rejects non-`none` quantization requests (see [Features](#features)).\n\n---\n\n## Use Cases\n\n### 1. **Chat Applications**\n\n```python\nimport requests\n\nmessages = []\nwhile True:\n user_input = input(\"You: \")\n messages.append({\"role\": \"user\", \"content\": user_input})\n r = requests.post(\n \"http://localhost:8000/v1/chat/completions\",\n json={\"model\": \"TinyLlama/TinyLlama-1.1B-Chat-v1.0\", \"messages\": messages},\n timeout=300,\n )\n content = r.json()[\"choices\"][0][\"message\"][\"content\"]\n messages.append({\"role\": \"assistant\", \"content\": content})\n print(f\"Assistant: {content}\")\n```\n\n### 2. **Scripted Batch Processing**\n\n```python\nimport requests\n\nprompts = [\"Summarize: ...\", \"Translate: ...\"]\nresults = [\n requests.post(\n \"http://localhost:8000/v1/completions\",\n json={\"model\": \"TinyLlama/TinyLlama-1.1B-Chat-v1.0\", \"prompt\": p, \"max_tokens\": 64},\n timeout=300,\n ).json()[\"text\"]\n for p in prompts\n]\n```\n\n> A dedicated `ai_cluster` client SDK (Python/JS) is planned — today the API is plain HTTP.\n\n---\n\n## Advanced Use Cases\n\n### 1. **Multi-Agent Systems (Agent per GPU)**\n\nYou can assign specific models to specific workers to create specialized agents.\n- **Worker 1 (AMD)**: Loads `llama3-8b` for Coding tasks.\n- **Worker 2 (NVIDIA)**: Loads `qwen3-coder-32b` for Reasoning/Review tasks.\n\n**Configuration**:\nNo special config needed. Just route your load requests:\n```bash\n# Load Coding Agent on AMD Worker\ncurl -X POST http://localhost:8000/v1/models/load \\\n -H \"Content-Type: application/json\" \\\n -d '{\"model_name\": \"llama3-8b\", \"worker_id\": \"<id from GET /v1/workers>\"}'\n\n# Load Reasoning Agent on NVIDIA Worker\ncurl -X POST http://localhost:8000/v1/models/load \\\n -H \"Content-Type: application/json\" \\\n -d '{\"model_name\": \"qwen3-coder-32b\", \"worker_id\": \"<id from GET /v1/workers>\"}'\n```\n\n### 2. **Running Large Models (Future Optimization)**\n\n> **Note**: Tensor and Pipeline parallelism core algorithms are implemented in `worker/src/parallelism.rs` (functions `tensor_parallel_llama_prefill`, `tensor_parallel_llama_decode_step`, `pipeline_parallel_llama_forward`), but are not yet wired into the gRPC inference service. Currently, models must fit within the memory of a single GPU, or you can leverage Data Parallelism by deploying multiple identical model workers on the same machine.\n\nWhen fully implemented, you will be able to configure model splitting like this:\n1. **Configure**: Ensure `config/models.toml` allows sufficient GPUs (Planned feature).\n ```toml\n [models.llama3-70b]\n max_gpus = 8\n parallelism.default = \"tensor\"\n ```\n2. **Load**:\n ```bash\n curl -X POST http://localhost:8000/v1/models/load \\\n -H \"Content-Type: application/json\" \\\n -d '{\"model_name\": \"llama3-70b\", \"worker_id\": \"nvidia-gpu-0\"}'\n ```\nThe system will automatically allocate 4 GPUs and use **Tensor Parallelism** (faster) or **Pipeline Parallelism** (inter-node) based on topology.\n\n---\n\n## Use on Real Scenarios\n\nAICluster turns your own PCs into a private AI coding assistant. One PC (the\n**coordinator**) receives requests; the PCs with graphics cards (the **workers**)\nrun the AI models. Once it's running, coding tools like\n[Claude Code](https://docs.claude.com/en/docs/claude-code) on your laptop talk to\nit exactly like they'd talk to a cloud AI — but every request, and your code,\nstay on your own network.\n\nHere's the whole thing, end to end:\n\n**1. On a PC with a graphics card, start a worker.** It needs a small helper\nprogram (`llama-server`) that actually runs the models; the Docker image already\nincludes it, so `docker compose up -d --build` is all you need. (On a bare-metal\ninstall you build it once — see\n[Deployment → llama-server for agentic serving](docs/deployment.md).)\n\n**2. On the PC that will receive requests, start the coordinator.** With Docker\nCompose it comes up next to the worker. To require a password on every request,\nset one variable before starting (skip it to run with no password):\n\n```bash\nexport COORDINATOR_API_KEYS=your-secret-key\n```\n\n**3. From your laptop, check it works** — replace `<coordinator-host>` with the\ncoordinator PC's name or IP address:\n\n```bash\npython scripts/validate_agentic.py \\\n --base-url http://<coordinator-host>:8000 \\\n --api-key your-secret-key\n```\n\n**4. Point Claude Code at it.** On each laptop, copy these four lines into the\nterminal — changing `<coordinator-host>` to the coordinator PC's address — then\nrun `claude`:\n\n```bash\nexport ANTHROPIC_BASE_URL=http://<coordinator-host>:8000\nexport ANTHROPIC_AUTH_TOKEN=your-secret-key # any text if you skipped the password\nexport ANTHROPIC_MODEL=qwen3-coder-30b-a3b-instruct-gguf # the \"big\" model\nexport ANTHROPIC_SMALL_FAST_MODEL=devstral-small-2-24b-instruct-gguf # the \"quick\" model\nclaude\n```\n\nThat's it — Claude Code now uses your cluster. It can read files, run commands,\nand edit code (tool use) just as it does against the cloud, because the models\nabove are trained for exactly that.\n\n**Good to know:**\n\n- **The first question is slow, then it's fast.** The first time you use a model,\n its file (~15 GB) downloads and starts up automatically; after that it stays\n loaded and answers quickly. Running the check in step 3 first gets this out of\n the way.\n- **Your chat history stays on your laptop.** Claude Code keeps each conversation\n locally; the cluster just answers one request at a time and remembers nothing\n between them.\n- **The model name must match one from `config/models.toml`** (like the two\n above). Claude Code's built-in menu only lists cloud models, so pick yours with\n the `ANTHROPIC_MODEL` variables, not the in-app menu.\n- **Seeing `400` errors?** Add `export CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1`\n and restart `claude`.\n- **Using a different tool?** Setups for Cline, aider, Continue, and Open WebUI\n live in [docs/clients.md](docs/clients.md).\n\n---\n\n## Installation\n\n### Prerequisites\n\n- **OS**: Ubuntu 22.04+ (recommended), RHEL 9, Rocky Linux 9\n- **Python**: 3.10+\n- **Rust**: 1.70+\n- **Docker**: 20.10+ (optional)\n- **GPU Drivers**: ROCm 6.0+ (AMD) or CUDA 12.1+ (NVIDIA)\n\n### Method 1: From Source\n\n```bash\n# Clone repository\ngit clone https://github.com/caestrada1103/ai-cluster.git\ncd ai-cluster\n\n# Set up Python environment\npython -m venv venv\nsource venv/bin/activate\npip install -r coordinator/requirements.txt\n\n# Build Rust worker\ncd worker\ncargo build --release --features=rocm # For AMD\n# or\ncargo build --release --features=cuda # For NVIDIA\n\n# Return to root\ncd ..\n\n# Create model directory\nmkdir -p models\n\n# Start coordinator\n# From the repo root (module path matters — cd coordinator breaks imports):\nuvicorn coordinator.main:app --host 127.0.0.1 --port 8000\n\n# In another terminal (from the repo root — the binary is at the workspace\n# root target/, not worker/target/, since `worker` is a Cargo workspace member):\n./target/release/ai-worker --port 50051 --gpu-ids 0\n```\n\n### Method 2: Docker Compose\n\nSee [Quick Start → Method 1](#method-1-docker-compose) above for the full,\ncurrent command sequence (env vars required, GPU profile selection).\n\n```bash\ndocker compose up -d --build\ndocker compose logs -f\ndocker compose down\n```\n\n### Method 3: Kubernetes (planned)\n\nKubernetes manifests are not shipped yet. Deploy with Docker Compose (Method 2)\nor run the services natively. K8s manifests/Helm are tracked on the roadmap.\n\nFor detailed installation instructions, see the [Deployment Guide](docs/deployment.md).\n\n---\n\n## Configuration\n\n### Coordinator Configuration (environment variables)\n\nThe coordinator is configured exclusively via `COORDINATOR_*` environment\nvariables (or `.env`). Key settings:\n\n```bash\n# 0.0.0.0 requires COORDINATOR_API_KEYS to be set too — the coordinator refuses\n# to bind non-loopback with no auth configured. Use 127.0.0.1 for local-only.\nCOORDINATOR_HOST=0.0.0.0\nCOORDINATOR_PORT=8000\nCOORDINATOR_DISCOVERY_METHOD=static # static only today (mdns/consul planned)\nCOORDINATOR_STATIC_WORKERS=localhost:50051 # comma-separated host:port list\nCOORDINATOR_ROUTING_STRATEGY=least_load # least_load|round_robin|random|affinity|power_of_two\n```\n\nSee the [Configuration Guide](docs/configuration.md) for the full table.\n\n### Minimal Worker Configuration\n\n```toml\n# config/worker.toml — FLAT schema (unknown keys are rejected)\ngrpc_port = 50051\nmetrics_port = 9091\ngpu_ids = [0]\nmodel_cache_dir = \"./models\"\n# grpc_bind_host / llamaserver_bind_host both default to 127.0.0.1 — see\n# Security: Secure by Default above for the opt-in env vars.\n```\n\nFor complete configuration options, see the [Configuration Guide](docs/configuration.md).\n\n---\n\n## API Reference\n\n### Endpoints\n\n| Method | Endpoint | Description |\n|--------|----------|-------------|\n| GET | `/health` | Health check |\n| GET | `/v1/workers` | List workers |\n| POST | `/v1/workers/manual` | Manually register worker addresses |\n| GET | `/v1/models` | List models (OpenAI-style list) |\n| POST | `/v1/models/load` | Load a model |\n| DELETE | `/v1/models/{name}` | Unload a model |\n| POST | `/v1/completions` | Generate text (buffered) |\n| POST | `/v1/chat/completions` | OpenAI-compatible chat (supports SSE streaming) |\n| GET | `/metrics` | Prometheus metrics |\n\n### Example: Text Completion\n\n```bash\ncurl -X POST http://localhost:8000/v1/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"deepseek-7b\",\n \"prompt\": \"Once upon a time\",\n \"max_tokens\": 50,\n \"temperature\": 0.8\n }'\n```\n\nResponse:\n```json\n{\n \"request_id\": \"3fa85f64-5717-4562-b3fc-2c963f66afa6\",\n \"text\": \" in a faraway land, there lived a brave knight...\",\n \"tokens_generated\": 50,\n \"processing_time_ms\": 2140.5,\n \"worker_id\": \"worker-0\"\n}\n```\n\nFor complete API documentation, see the [API Reference](docs/api_reference.md).\n\n---\n\n## Monitoring\n\n### Prometheus Metrics\n\n```bash\n# Scrape metrics (Prometheus is on 9099, Coordinator on 8000)\ncurl http://localhost:8000/metrics\n\n# Example metrics\n# HELP coordinator_requests_total Total requests processed\n# TYPE coordinator_requests_total counter\ncoordinator_requests_total{model=\"deepseek-7b\"} 1250\n\n# HELP worker_gpu_utilization_percent GPU utilization\n# TYPE worker_gpu_utilization_percent gauge\nworker_gpu_utilization_percent{worker=\"worker-gpu-0\",gpu=\"0\"} 75.2\n```\n\n### Grafana Dashboards\n\nProvisioning ships in `monitoring/`: the Prometheus datasource plus an\n**AI Cluster Overview** dashboard (request rate, P95 latency, GPU\nutilization/temperature, coordinator backlog). Grafana: http://localhost:3000\n(admin / $GRAFANA_ADMIN_PASSWORD).\n\n### Distributed Tracing (planned)\n\nJaeger integration is on the roadmap — no tracing ships today.\n\n---\n\n## Contributing\n\nWe welcome contributions! Please see our [Contributing Guidelines](CONTRIBUTING.md).\n\n### Development Setup\n\n```bash\n# Clone your fork\ngit clone https://github.com/caestrada1103/ai-cluster.git\ncd ai-cluster\n\n# Set up pre-commit hooks\npip install pre-commit\npre-commit install\n\n# Run tests (unit suite lives under coordinator/)\npytest coordinator/\ncd worker && cargo test --features wgpu\n```\n\n### Development Workflow\n\n1. Fork the repository\n2. Create a feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n### Code Style\n\n- **Python**: Black with line length 100\n- **Rust**: rustfmt with default settings\n- **Documentation**: Markdown with linter\n\n---\n\n## License\n\nThis project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.\n\n```\nCopyright 2026 AI Cluster Contributors\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n```\n\n---\n\n## Acknowledgments\n\n### Built With\n\n- [Burn](https://burn.dev/) - Rust deep learning framework\n- [ROCm](https://rocm.docs.amd.com) - AMD GPU computing platform\n- [CUDA](https://developer.nvidia.com/cuda-toolkit) - NVIDIA GPU computing platform\n- [FastAPI](https://fastapi.tiangolo.com/) - Python web framework\n- [PyTorch](https://pytorch.org/) - For model conversion\n- [HuggingFace](https://huggingface.co) - Model hub and tokenizers\n\n### Inspired By\n\n- [vLLM](https://vllm.readthedocs.io/) - PagedAttention and continuous batching\n- [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) - Tensor parallelism\n- [DeepSpeed](https://www.deepspeed.ai/) - Pipeline parallelism\n- [NVIDIA Dynamo](https://github.com/NVIDIA/dynamo) - Distributed inference\n\n### Contributors\n\n<a href=\"https://github.com/caestrada1103/ai-cluster/graphs/contributors\">\n <img src=\"https://contrib.rocks/image?repo=caestrada1103/ai-cluster\" />\n</a>\n\n### Support\n\n- 📚 [Documentation](docs/)\n\n---\n\n## Star History\n\n[](https://star-history.com/#caestrada1103/ai-cluster&Date)\n\n---\n\n<div align=\"center\">\n <sub>Built with ❤️ by the AI Cluster Team</sub>\n <br>\n <sub>© 2026 AI Cluster. All rights reserved.</sub>\n</div>\n"}}}} |